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ABSTRACT 


Part  I  of  The  Grade  IX  Social  Studies  Departmental  of  19 66  was 
comprised  of  one  hundred  multiple  choice  items  based  on  "Bloom's 
Taxonomy."  The  purpose  of  this  investigation  was  to  ascertain  whether 
or  not  this  examination  showed  evidence  of  sampling  a  process  dimension 
of  evaluation  0  Specifically,  examination  results  were  subjected  to  tests 
of  construct  and  concurrent  validity  to  determine  whether  empirical  evi¬ 
dence  could  be  found  that  this  examination  required  the  use  of  higher 
mental  processes  as  distinct  from  the  ability  to  recall  content  material. 

A  randomly  selected  sample  of  nine  hundred  and  fifty-eight  sets 
of  examination  scores  provided  the  major  portion  of  data  for  the  investi¬ 
gation.  Construct  validity  was  examined  through,  first,  having  the  test 
items  classified  into  Taxonomy  Categories  by  judges  and  comparing  this 
classification  to  that  of  the  committee  designing  the  examination.  Second, 
the  statistical  technique  of  factor  analysis  was  applied  to  the  test 
results .  Test  items  were  analyzed  individually  and  in  groups  to  attempt 
to  determine  whether  factor  patterns  could  be  found  which  verified  the 
classification  of  items  into  recall  and  higher  mental  process  categories. 
Concurrent  validity  was  examined  by  grouping  the  items  of  Part  I  of  the 
examination  into  recall  and  high  mental  process  subtests  on  the  basis  of 
the  judge  and  committee  classifications  and  then  correlating  these  sub¬ 
tests  with  certain  external  criteria  to  determine  whether  they  related  in 
a  significantly  different  manner  to  these  criteria.  The  external  criteria 
utilized  were  the  Principal's  Ratings,  scores  on  the  School  and  College 
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Ability  Test,  and  the  scores  obtained  on  Part  II,  the  subjectively- 
scored,  written  section  of  the  examination. 

It  was  found  that  the  hypothesis,  postulating  agreement  among 
judges,  was  supported.  Hypotheses  further  related  to  the  construct 
validity  of  the  test  were  not  supported.  Factor  analysis  revealed  one 
major  factor  accounting  for  a  large  proportion  of  the  test  variance. 
Examination  of  various  factor  solutions,  including  orthogonal  and  oblique 
rotations  did  not  reveal  results  which  could  clearly  be  interpreted  in 
terms  of  the  process  dimension  the  test  purported  to  measure.  It  was 
found  that  the  hypothesis  postulating  a  significant  difference  in  the 
relationship  of  external  criteria  to  recall  and  higher  mental  process 
subtests  was  supported  in  two  cases.  Principal's  Ratings  and  Part  II 
Scores  correlated  significantly  higher  with  the  recall  subtests  than  with 
higher  mental  process  subtests .  School  and  College  Ability  Test  scores 
did  not  differ  significantly  in  correlating  with  subtests. 
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CHAPTER  I 


STATEMENT  OF  THE  PROBLEM 

I .  INTRODUCTION 

The  1966  Grade  IX  Social  Studies  Departmental  Examination  represented 
a  pioneer  effort  to  introduce,  in  an  empirically  verifiable  way.,  a  thought 
process  dimension  into  provincial  examinations  in  the  social  studies. 

In  his  report  to  the  Cameron  Commission  R.  S.  MacArthur  recommended 
that  "continued  and  increased  effort  be  made  to  construct  objectively 
scored  test  items  which  test  such  thought  processes  as  understandings  and 
appreciations  in  addition  to  factual  knowledge."*  As  part  of  a  continuing 
program  to  implement  such  recommendations  and  thus  make  departmental  exam¬ 
inations  more  reflective  of  course  objectives,  The  1966  Grade  IX  Social 
Studies  Departmental  Examination  was  designed  to  examine  the  attainment  of 
objectives  requiring  the  use  of  intellectual  skills  and  abilities  as  dis¬ 
tinct  from  those  objectives  requiring  recall  ability. 

To  provide  a  structure  for  their  work,  those  individuals  selected 

by  The  Alberta  High  School  Entrance  Examinations  Board  to  set  the  examina- 

2 

tion  used  Bloom  *  s  Taxonomy  of  Educational  Objectives :  Cognitive  Domain, 

The  rationale  for  the  test  -was  presented  in  the  following  manner; 

^R<.  S.  MacArthur  and  S.  Hunka,  School  Examination  Practices  and 
Standards  in  Alberta  (The  Alberta  Advisory  Committee  on  Educational  Re¬ 
search,  University  of  Alberta,  i960)  pp.  40-4l. 

B.  S.  Bloom,  Taxonomy  of  Educational  Objectives ;  Cognitive 
Domain  (New  York:  David  McKay  Company,  19657*. 
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"The  acquisition  of  knowledge,"  "reaching  understandings," 

'the  achievement  of  basic  skills,"  "critical  thinking  ability," 
these  are  expressions  commonly  found  in  the  literature  stating 
objectives  of  the  social  studies.  What  precisely  do  they  mean? 
Are  they  discreet  components  of  the  learning  process?  Can  in¬ 
creased  clarity  be  given  to  these  expressions? 

Addressing  themselves  to  questions  such  as  these.  Bloom  and 
his  associates  published  The  Taxonomy  of  Educational  Objectives : 
Cognitive  Domain  in  which  educational  objectives  are  classified 
with  increased  clarity  and  precision.  In  so  doing  thev  not  only 
add  a  dimension  of  specificity  to  guide  teachers  in  defining 
objectives,  but  have  also  enableu  cest  builders  to  measure  more 
effectively.  It  is  essentially  for  this  reason  that  "Bloom's 
Taxonomy"  has  been  used  as  a  guideline  in  the  task  of  construct¬ 
ing  the  Grade  IX  Social  Studies  Examination. 3 

Working  from  this  theoretical  basis,  the  committee  constructing 
the  examination  developed  one  hundred  objective  questions  of  the  multiple 
choice  type  to  form  Part  I  of  the  examination.  Forty  of  these  questions 
were  designed  to  require  various  types  of  recall  by  the  students.  The 
remaining  questions  were  designed  to  require  the  utilization  of  the  in«- 
telleetual  abilities  and  skills  which  comprise  the  upper  levels  of  The 
Taxonomy.  Figure  1  is  a  "blueprint"  showing  the  relative  weight  given 
to  various  areas  of  content  and  ability  on  the  objective  section  of  the 
examination. 

These  questions  provided  one  hundred  of  a  total  of  one  hundred 
and  thirty  marks  on  the  examination.  The  remaining  thirty  marks  were 
allotted  for  written  answers  to  a  number  of  completion  and  short  essay 
type  questions  contained  in  Part  II  of  the  examination.  Subjective 
questions  were  intended  to  examine  student  ability  to  draw  on  factual 
knowledge  and  to  manipulate  this  knowledge  through  higher  mental  thought 


^Summary  Description  of  Grade  Nine  Social  Studies  Objectives 
and  Test  Items  (The  High  School  Entrance  Examination  Board,  Alberta 
Department  of  Education,  19 66),  p.  4. 
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processes. 


This  study  was  undertaken  to  determine  whether  or  not  the  results 
obtained  on  the  examination  could  be  used  to  verify  the  division  of  the 
objective  items  into  recall,  and  intellectual  ability  and  skill  categorie 

II.  PROBLEM 

This  investigation  seeks  to  answer  the  following  question:  Do 
the  results  obtained  by  a  randomly  selected  sample  of  students  on  this 
examination,  when  subjected  to  tests  of  construct  and  concurrent  validity 
show  evidence  that  the  examination  did  require  the  use  of  intellectual 
abilities  and  skills  as  distinct  from  the  ability  to  recall  factual 
material? 

III.  NEED  FOR  THE  STUDY 

Contrary  to  the  objectives  of  educators,  many  currently  used  tests 
have  tended  to  emphasize  tasks  requiring  only  the  recall  of  specific 
content  material.  Others  do  require  additional  abilities  but  in  many 
cases  no  assessment  of  these  requirements  has  even  been  made.  In  order 
that  an  instrument  of  evaluation  may  accurately  reflect  the  goals  and 
purposes  of  the  educational  process,  test  situations  should  offer  the 
student  the  opportunity  to  use  various  thought  processes  and  guarantee 
that  performance  in  these  areas  will  be  recognized.  "A  major  problem 
of  educational  measurement  has  been  finding  ways  and  means  of  multiply¬ 
ing  the  measurable  dimensions  of  educational  achievement  to  the  point 
where  they  come  as  close  as  possible  to  encompassing  all  the  important 
categories  of  behaviour  that  are  likely  to  be  acquired  inside  and  outside 
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the  classroom."  In  connection  with  this  basic  problem  Bloom  stated,  "The 
most  cursory  reading  of  the  tests  available .. .would  indicate  that  tremen- 

5 

dous  emphasis  is  given  in  our  schools  to  this  kind  of  remembering  or  recall." 

Further  documentation  is  provided  by  the  discussion  of  purpose  found 
in  curriculum  guides  and  testing  manuals.  Creators  of  tests  are  apparently 
cognizant  of  the  desirability  of  constructing  items  which  call  for  the  use 
of  thought  processes  in  addition  to  the  recall  of  factual  material.  How¬ 
ever,  as  yet  few  attempt  to  do  this.  A  review  of  the  Stanford  Achieve¬ 
ment  Test  indicated  that,  "It  is  hard  to  see  how  the  first  two  of  these 
tests  (Social  Studies  and  Science)  reflect  the  trend  towards  the  teaching 
of  meanings  and  understandings  rather  than  mere  factual  knowledge  to 
which  the  authors  pay  their  respects  in  the  manual. . .these  tests  still 

deal  with  miscellaneous  knowledge  rather  than  with  problem-solving  skills, 

6 

critical  understandings  and  applications  of  learning." 

The  fact  that  many  tests  measure  only  recall  processes  presented 
a  second  major  problem.  Because  of  his  perception  of  what  the  test  calls 
for,  the  activities  the  student  engages  in,  in  preparation  for  a  test, 
will  be  centered  around  memorizing  material,  rather  than  understanding 


henry  S.  Dyer,  "Educational  Measurement --Its  Nature  and  Problems," 
Evaluation  in  The  Social  Studies,  Thirty-Fifth  Yearbook  of  The  National 
Council  for  the  Social  Studies  (Washington,  D.  C.  Department  of  National 
Education  Association,  1965)*  p.  4l. 

^Benjamin  S.  Bloom  (ed. )  Taxonomy  of  Educational  Objectives  Hand¬ 
book;  I  Cognitive  Domain  (New  York;  David  McKay  Co.  Inc .  1961 ) ,  p .  30. 

c 

°0scar  K. Buros  (ed.).  The  Fifth  Mental  Measurements  Yearbook 
(Highland  Park,  New  Jersey:  The  Gryphon  Press,  1959 )>  PP-  78-80. 
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concepts  and  ideas.  Merkhofer  indicated  that  the  extent  to  which  students 
try  to  memorize  material  in  contrast  to  the  development  of  problem  solving 
skills  directed  toward  the  application  of  ideas  and  methods  tends  to  be 
directly  related  to  the  nature  and  type  of  examination  the  students  expect 
to  receive. 

A  third  aspect  of  this  problem  involved  most  teachers  being  aware 
of  the  objectives  being  tested  by  various  tests,  tending  to  stress  these 
kinds  of  learning  experiences.  Tyler  reported  that  The  Regent ' s  Inquiry 
into  The  Character  and  Cost  of  Public  Education  in  New- York  State  found 
that  teachers,  "were  conscious  of  the  objectives  being  tested  in  the  exam¬ 
inations  and  sought  to  emphasize  these  kinds  of  learning  in  their  classes 
rather  than  to  follow  the  objectives  recommended  in  the  local  curriculum 

Q 

guides . ” 

The  preceding  studies  suggested  that  when  the  items  in  an  achieve¬ 
ment  test  encourage  the  student  to  use  his  intellectual  abilities,  the 
test  itself  will  have  a  constructive  influence  on  the  learning  process. 
However,  if  the  major  emphasis  of  an  achievement  test  is  placed  upon  rote 
learning  and  recall  of  specific  facts,  the  test  may  act  as  a  deterrent 
to  learning  experiences. 

This  is  not  to  say  that  factual  knowledge  is  not  important. 

7 

B.  E.  Merkhofer,  "College  Students  Study  Behavior"  unpublished 
Ph.  D.  dissertation,  University  of  Chicago,  1963). 

^Ralph  W.  Tyler,  The  Regent's  Inquiry  into  the  Character  and  Cost 
of  Public  Education,"  Invitational  Conference  on  Testing  Problems  -  1959  ~ 
Proceedings  (Princeton,  New  Jersey:  Educational  Testing  Service,  19^0 ) , 

p.  11. 
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Whitehead  and  Dressel  would  indeed  agree  that  the  acquisition  of  know¬ 
ledge  is  an  important  outcome  of  instruction.  They  would  also  recognize, 
however,  that  learning  experiences  must  provide  the  student  with  the 
ability  to  integrate  and  apply  material  which  otherwise  might  remain  iso¬ 
lated.  Methods  of  evaluation  must,  therefore,  be  designed  to  reinforce 
the  educational  processes  which  increase  the  student’s  ability  to  handle 
problems  by  analyzing  or  understanding  the  new  situation,  by  applying  a 
whole  body  of  knowledge  which  is  readily  available,  and  by  relating  ap¬ 
propriate  elements  of  past  experience. 

These  aforementioned  problems  are  particularly  apparent  in  evalu¬ 
ating  the  social  studies.  Engle  revealed  the  basis  for  these  problems 
as  he  pointed  out  the  importance  of  decision  making. 

...In  teaching  the  social  studies,  we  should  emphasize 
decision  making  as  against  mere  remembering.  We  should 
emphasize  decision  making  at  two  levels:  at  the  level 
of  deciding  what  a  group  of  descriptive  data  means,  how 
these  data  may  be  summarized  and  generalized,  what  princi¬ 
ples  they  suggest;  and  also  decision  making  at  the  level  of 
policy  determination,  which  requires  a  synthesis  of  facts, 
principles  and  values  usually  not  all  found  on  one  side  of 
any  question.  1 

In  the  specific  case  studied  the  same  basic  set  of  objectives  was  held, 
the  attainment  of  which,  if  they  were  to  be  measured,  required  the  assess 
ment  of  students'  higher  level  cognitive  processes.  The  Junior  High 


9 

Alfred  N.  Whitehead,  The  Aims  of  Education  and  Other  Essays 
(Netf  York:  The  New  American  Library,  19&1 ) . 

1C)Paul  L.  Dressel  and  Associate,  Evaluation  in  Higher  Education 
(Boston:  Houghton  Mifflin  Co. ,  1961). 

^S.  H.  Engle,  "Decision  Making:  The  Heart  of  Social  Studies 
Instruction,"  Crucial  Issues  in  The  Teaching  of  the  Social  Studies 
(Englewood  Cliffs,  New  York:  Prentice  Hall  Inc.,  1964),  p.  28. 
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School  Curriculum  Guide  presents  the  grade  nine  social  studies  course  ob¬ 
jectives  in  this  way. 

In  this  course ,  objectives  for  each  unit  are  listed  under 
three  headings' — (l)  understandings,  (2)  skills,  abilities 
and  habits,  (3)  attitudes.  This  does  not  mean  that  factual 
knowledge  is  not  required  for  generalizations  are  based  on 
facts.  It  is  well  known  that  facts  elude  the  memory  if  they 
are  not  being  used  constantly,  but  a  generalization,  which  is 
the  result  of  reasoning,  is  an  enduring  gain  which  may  be 
used  in  further  critical  thinking.^ 

If  the  attainment  of  objectives  requiring  the  use  of  intellectual 
skills  and  abilities  can  be  shown  to  be  a  distinct  and  empirically 
recognizable  part  of  an  examination,  a  significant  step  will  have  been 
taken  towards  solving  these  problems  in  evaluation.  If  investigation  of 
the  results  of  using  this  testing  instrument  shows  evidence  that  it 
measures  attainment  in  the  area  of  higher  mental  processes  and  can  dis¬ 
tinguish  this  attainment  from  mere  recall,  then  instruction  may  be  allied 
more  closely  to  course  objectives  in  Grade  IX  Social  Studies. 

Several  studies  have  dealt  with  the  measurement  of  intellectual 
processes.  However,  as  Crawford1^  points  out  these  projects  have  been 
conducted  under  artificial  conditions.  Test  items  are  usually  specially 
constructed  by  researchers  involved  in  a  special  project  rather  than  by 
public  educators  as  part  of  the  regular  school  program.  An  example  of 
such  work  is  found  in  the  final  report  of  a  U.  S.  0.  S.  project  completed 


“i  r> 

"^Department  of  Education  of  Alberta,  Junior  High  School  Curriculum 
Guide  for  Social  Studies  -  Language  (1963  Edition) . 

■^William  R.  Crawford,  ’’The  Empirical  Validity  of  Items  Constructed 
to  Measure  Complex  Intellectual  Processes,"  (an  unpublished  paper  presented 
to  the  1967  Annual  Meeting  of  The  American  Educational  Research  Associa¬ 
tion). 
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by  Kropp,  Stoker  and  Bashaw.1^  This  type  of  work  is  useful  in  terms  of 
basic  research,  but  it  provides  little  evidence  with  respect  to  the  out¬ 
comes  which  could  be  expected  if  items  were  constructed  by  practising 
educators  and  were  part  of  regular  examination  procedures  as  was  the  case 
in  the  construction  and  administration  of  The  19 66  Grade  IX  Social  Studies 
Departmental  Examination » 


IV *  HYPOTHESES 

The  basic  problem  of  the  study  was  approached  by  applying  tests  of 
construct  and  concurrent  validity  to  the  examination  and  the  examination 
results . 

Construct  Validity.  Investigation  of  the  construct  validity  of  the 
examination  was  based  on  the  following  hypotheses. 

Hypothesis  (Al)  A  group  of  twelve  judges,  judging  independently, 
will  reach  agreement  in  classifying  the  objective  items  of  the 
test  into  knowledge  and  higher  mental  process  categories. 

Hypothesis  (A2)  The  responses  to  the  objective  items  classified 
in  each  Taxonomy  category  will  correlate  positively. 

Hypothesis  (A3)  Factor  analysis  of  correlations  among  items  will 
result  in  higher  mental  process  items  loading  on  factors  in  a  man¬ 
ner  distinct  from  the  loading  of  recall  items. 

Concurrent  Validity.  Investigation  of  the  concurrent  validity  of 


P.  Kropp,  H.  W.  Stoker  and  W.  H.  Bashaw,  The  Construction  and 
Validation  of  tests  of  the  cognitive  processes  as  described  in  The 
Taxonomy  of  Educational  Objectives .  U.  S.  Office  of  Educational  Co¬ 
operative  Research-Project  No.  2117.  Tallahassee,  Florida:  Florida 
State  University,  1966. 
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the  examination  was  based  on  the  following  hypotheses. 

Hypothesis  (Bl)  A  positive  correlation  v/ill  be  obtained  between 
scores  on  objective  test  items  identified  as  requiring  only  recall 
and 

(a)  SCAT  Total  Scores 

(b)  SCAT  Verbal  Sub-Scores 

(c)  SCAT  Quantitative  Sub-Scores 

(d)  Principals'  Ratings 

(e)  Examination  Sub-Scores  from  the  Essay  Section, 

Part  II  of  The  Examination 

Hypothesis  (B2)  A  positive  correlation  will  be  obtained  between 
scores  on  objective  test  items  identified  as  requiring  higher 
mental  processes  and  the  criteria  presented  in  hypothesis  Bl. 
Hypothesis  (B3)  Correlations  obtained  while  testing  hypothesis 
Bl  will  differ  significantly  from  correlations  obtained  while 
testing  hypothesis  B2. 


V.  ASSUMPTIONS 


The  following  statements  of  Henry  S.  Dyer  and  Robert  ¥.  McFall 

outlined  the  assumptions  on  which  the  preceding  hypotheses  were  based. 

Assumption  One  (relevant  to  hypotheses  Al,  A2,  and  A3») 

The  construct  validity  of  a  test  may  be  determined  by  three 
steps: 

(a)  First,  can  a  group  of  any  twenty  examiners,  judging  inde¬ 
pendently,  agree  reasonably  well  on  classification  of  responses? 

(b)  Do  the  responses  classified  as  one  or  the  other  tend  to 
hang  together  empirically?  That  is,  do  the  responses  rank 
childreh  in  the  same  order? 

(c)  Third,  is  the  agreement  in  rank  order  within  one  set  of 
responses  distinguishably  greater  on  the  average  than  the  agree- 


' 
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ment  between  different  sets  of  responses?  If  the  answers  to  all 
three  questions  are  in  the  affirmative,  then  it  can  be  said  that 
the  test  is  measuring  these  constructs.1^ 

Assumption  Two  (relevant  to  hypotheses  Bl,  B2,  and  B3) 

Concurrent  validity  will  be  established  if: 

(a)  There  is  a  positive  relationship  between  scores  on  test  items 
which  involve  only  the  recall  of  specific  area  of  study. 

(b)  There  is  a  low  relationship  between  scores  on  test  items 
which  do  not  involve  the  recall  of  specific  content  material  and 
current  methods  of  evaluating  student  achievement  in  a  specific 
area  of  study. 

(c)  The  relationship  between  items  requiring  only  the  recall  of 
specific  content  material  and  the  relationship  between  test  items 
which  do  not  require  the  recall  of  specific  content  material  and 
the  criterion  variables  are  significantly  different. 


VI.  DELIMITATIONS 


Consideration  was  given  only  to  examining  the  dichotomous  recall- - 
higher  mental  process  facet  of  the  examination  in  terms  of  construct  and 
concurrent  validity. 

(1)  No  attempt  was  made  to  assess  the  level  of  attainment  of  any 
individual  or  group  of  individual  marks. 

(2)  No  comparisons  were  made  among  any  parts  of  the  sample 
population. 

(3)  No  attempt  was  made  to  produce  statistical  evidence  to  justify 
or  identify  the  entire  hierarchy  of  Bloom's  Taxonomy. 

(4)  No  assessment  was  made  of  the  examination's  content  validity. 


1^Henry  S.  Dyer,  "Educational  Measurement — Its  Nature  and  Problems 
Evaluation  in  the  Social  Studies  (Washington,  D.  C.,  N.C.S.S.,  1965)*  p. 

^Robert  W.  McFall,  "The  Development  of  Validation  of  an  Achieve¬ 
ment  Test  for  Measuring  Higher  Level  Cognitive  Processes  in  General 
Science,"  The  Journal  of  Experimental  Education  (XXXIIl),  Autumn,  1964), 

pp.  IO3-IO6. 
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(5)  No  value  judgment  was  attempted  with  respect  to  the  relative 
merits  of  the  examination. 


VII.  DEFINITION  OF  TERMS 

Cognitive  Domain — "those  objectives  which  deal  with  the  recall  or 
recognition  of  knowledge  and  the  development  of  intellectual  skills  and 
abilities . 

Concurrent  Validity- -"is  concerned  with  the  relation  of  test  scores 
to  an  accepted  contemporary  criterion  of  performance  on  the  variable  which 
the  test  is  intended  to  measure." 

Construct  Validity- -is  concerned  with  "what  psychological  qualities 

a  test  measures"  and  is  evaluated  by  "demonstrating  that  certain  explana- 

„19 

tory  constructs  account  to  some  degree  for  performance  on  the  test. 

Content  Validity- -"is  concerned  with  the  adequacy  of  sampling  of  a 

20 

specified  universe  of  content." 

Higher  Level  Cognitive  Processes — those  processes  of  thought  which 
involve  using  an  additional  mental  ability  in  utilizing  recalled  material. 

Judge — someone  deemed  to  be  familiar  with  the  grade  nine  social 
studies  and  with  The  Taxonomy. 

Knowledge --as  an  objective  defined  by  Bloom,  involves  "the  recall 

l^Bloom,  0£.  cit . ,  p.  7. 
l  A 

xoRobert  L.  Ebel,  Measuring  Educational  Achievement  (New  York: 
Prentice-Hall  Inc.,  1965),  p.  380* 

19Ibid. 

20Ibid. 
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of  specifics  and  universals,  the  recall  of  methods  and  processes,  or  the 
recall  of  a  pattern  structure  or  setting.  For  measurement  purposes  the  re¬ 
call  situation  involves  little  more  than  bringing  to  mind  the  appropriate 
material. 

Principals'  Ratings --those  grades  recommended  by  principals  to  The 
Department  of  Education  for  those  students  intending  to  write  the  Grade  IX 
Social  Studies  Departmental  Examination. 

SCAT  Scores — the  scores  obtained  by  students  on  the  School  and 
College  Ability  Test  administered  in  Grade  IX. 

The  Taxonomy — refers  to  Bloom 1 s  Taxonomy  of  Educational  Objectives : 

C ognit ive  Domain.^ 

VIII o  ORGANIZATION  OF  THE  STUDY 

This  Chapter  contains  a  statement  of  the  problem,  a  discussion  of 
its  significance,  and  an  outline  of  its  hypothetical  basis  in  the  study. 
Also  stated  and  defined  were  the  delimitations  of  and  terms  essential  for 
understanding  the  study.  The  remainder  of  the  investigation  is  organized 
in  the  following  way: 

Chapter  II  contains  a  survey  of  related  literature.  The  first  part 
of  the  chapter  is  a  documentation  of  the  basic  problem  of  making  achieve¬ 
ment  tests  more  reflective  of  course  objectives.  Following  this,  methods 
of  approaching  the  problem  are  reviewed.  Finally,  research  on  the  identi¬ 
fication  of  mental  processes  and  the  use  of  The  Taxonomy  is  summarized. 

Chapter  III  contains  a  description  of  the  procedure  followed  and 

^^Bloom,  o£.  cit . ,  p.  48. 

22Ibid. 
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statistics  used  in  studying  the  construct  and  concurrent  validity  of  the 
examination. 

Chapter  IV  contains  the  findings  of  the  investigating  procedure 
described  in  Chapter  III .  The  first  part  of  the  chapter  deals  with  the 
results  of  attempting  to  establish  construct  validity  and  the  second  deals 
with  the  results  of  attempting  to  establish  concurrent  validity. 

Chapter  V  contains  conclusions,  implications,  and  suggestions  for 
further  research  arrived  at  from  the  findings  of  Chapter  IV. 
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CHAPTER  II 


REVIEW  OF  THE  LITERATURE 

INTRODUCTION 

No  previous  Alberta  Departmental  Examination  in  the  Social  Studies 

had  the  same  physical  structure  or  theoretical  basis  as  that  of  19 66. 

The  1965  Science  IX  Departmental  Examination  was  primarily  a  multiple- 

choice  objective  examination  based  on  "The  Taxonomy"  but  no  follow-up 

investigation  of  the  results  of  its  use  has  yet  been  made  public. 

12  3  4 

Bodnaruk,  Eddy,  Sabey,  and  Truckey  did  research  on  recent  Grade  IX 
Social  Studies  Departmental  but  these  studies  were  of  a  comparative  and 
local  nature.  Studies  which  would  be  equated  to  studying  the  validity 
of  an  application  of  The  Taxonomy  to  a  Departmental  Examination  on  social 
studies  were  non-existent. 

However,  the  construction  of  this  examination  and  the  study  of  its 
construct  and  concurrent  validity  was  representative  of  an  effort  to  ap- 


^Bodnaruk,  "A  Comparative  Study  of  Examination  Results,  Grades  IX 
and  XII,  in  One  Town  School  and  In  Three  Rural  Centralized  Schools  in  The 
County  of  Ponoka"  (unpublished  Master's  Thesis,  The  University  of  Alberta, 
Edmonton,  1962). 


P.  Eddy,  "A  Study  of  Certain  Characteristics  of  Teachers  in 
Relation  to  Grade  IX  Social  Studies"  (unpublished  Master's  Thesis,  The 
University  of  Alberta,  Edmonton,  1966). 


3r.  H.  Sabey,  "A  Comparison  of  The  Achievement  of  Grade  IX  Pupils 
In  Various  School  Organizations"  (unpublished  Master's  Thesis,  The 
University  of  Alberta,  Edmonton,  1966). 

L.  A.  Truckey,  "Comparison  of  Achievement  of  Grade  Nine  Students 
in  Selected  Single  Grade  and  Multi-Grade  Classes  in  Alberta"  (unpublished 
Master's  Thesis,  The  University  of  Alberta,  Edmonton,  1964). 
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proach  the  frequently  publicized  problem  of  making  mass  evaluative  instru 
ments  more  reflective  of  the  objectives  they  purport  to  measure.  "Bloom1 
Taxonomy”  represented  one  effort  of  educational  psychologists  to  classify 
educational  objectives  in  such  a  way  that  a  much  needed  process  dimension 
was  added.  A  limited  number  of  recent  studies  have  revealed  that  The 
Taxonomy  is  finding  application  in  efforts  to  add  a  mental  process  dimen¬ 
sion  to  testing  in  various  subject  fields. 


II.  EDUCATIONAL  OBJECTIVES  AND  MASS  EVALUATION 


With  the  phenomenal  expansion  of  public  education  and  the  conse¬ 
quent  expansion  of  mass  testing  procedures  there  has  developed  a  concern 
for  keeping  evaluative  procedures  reflective  of  course  objectives.  As 
Thorndike  states: 


The  process  of  measurement  of  achievement  is  secondary 
to  that  of  defining  educational  objectives.  The  ends  to  be 
achieved  must  be  formulated  clearly.  Then  measurement  pro¬ 
cedures  can  be  sought  as  tools  for  appraising  the  extent 
to  which  these  ends  have  been  achieved.'5 


Lindquist,  writing  of  the  same  problem,  described  the  direction 
in  which  the  measurement  of  objectives  tends  to  be  perverted. 

Where  no  authoritative  statements  of  immediate  objectives 
have  been  available  and  where  such  objectives  have  not  been 
sufficiently  specific  or  meaningful,  the  test  constructor 
has  often  set  up  or  derived  his  own  immediate  objectives  for 
the  test.  He  has  usually  derived  these  objectives,  however, 
not  from  any  general  ultimate  objectives,  but  from  the  common 
content  of  current  instruction. 


^R.  L.  Thorndike  and  E.  Hagen,  Measurement  and  Evaluation  in 
Psychology  and  Education  (New  York:  John  Wiley  and  Sons,  1961),  p.  1. 

°E.  F.  Lindquist,  Educational  Measurement ,  (Washington,  D.  C.: 
American  Council  on  Education,  1950),  p.  133* 
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He  further  identified  the  source  of  the  problem  in  the  process  of 
test  construction. 

Analysis  of  testbooks  and  courses  of  study  has  contributed 
the  most  common  technique  for  validating  educational  achieve¬ 
ment  tests.  Accordingly,  just  as  the  real  objective  of  in¬ 
struction  has  been  to  teach  what  is  in  the  textbook,  so,  in 
many  instances  the  real  objective  in  testing  has  to  an  even 
greater  degree  been  "to  test  for  recall  of  what  is  in  the  text¬ 
books,  or  to  test  what  is  now  being  taught.  " 

Downey  reiterated  this  position  when  he  stated  that  the  practice 
of  testing  only  a  student's  mastery  of  factual  material  was  a  "vicious" 
thing  and  would  only  lead  to  "the  distortion  of  the  educative  process 

Q 

if  carried  on  intensively." 

Engle, ^  Lewenstein,  and  Massialas  have  pointed  out  that  the  problem 
is  particularly  apparent  in  evaluation  of  the  social  studies.  Lewenstein 
states,  "the  recall  of  fact  should  be  distinguished  from  behaviours  which 
indicate  understanding,  the  ability  to  show  insight,  to  make  new  associ¬ 
ations,  or  to  apply  concepts  to  new  situations  which  the  respondents 
have  never  faced  before.  Massialas  summarized  the  present  position  of 
social  studies  testing  when  noting  first,  that  there  is  a  lack  of  research 
dealing  with  evaluation  in  the  social  studies  and  second,  that  most  of  the 
tests  and  manuals  that  accompany  social  studies  textbooks  include  ques- 


7lbid. ,  p .  130 . 

O 

°L.  W.  Downey,  Introductory  Address  to  an  Institute  on  Testing, 
Lacombe,  Alberta,  1964,  p.  3* 

^S.  H.  Engle,  "Objectives  of  The  Social  Studies,"  New  Challenges 
in  the  Social  Studies  (Belmont,  California:  Woodsworth  Publishing  Co., 
196517 

l°Morris  R.  Lewenstein,  Teaching  Social  Studies  in  Junior  and  Senior 
High  Schools  (Chicago:  Rand  McNally  and  Co.,  1965),  P°  462. 
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tions  which,  generally,  require  only  recall. 

12 

Davies  and  Tinsley  analyzed  the  classroom  communication  of  social 
studies  teachers.  Theii  findings  supported  the  generalizations  of 
Massialas  and  Lewenstein  as  being  applicable  to  the  classroom  situation  as 
well  as  to  tests  and  textbooks.  Results  of  the  study  indicated  that  most 
questions  asked  by  social  studies  teachers  are  procedural  in  nature  or  re¬ 
quire  pupil  memory  and  that  higher  order  objectives  are  seldom  behaviour- 
ally  operational. 

III.  THE  CLASSIFICATION  OF  OBJECTIVES 


Authorities  generally  seemed  to  agree  that  solving  this  problem 
must  begin  with  a  clear  classification  of  objectives.  As  Ebel  has  stated 
’’part  of  the  reluctance  of  teachers  to  base  their  tests  on  course  object¬ 
ives  has  been  due  to  inadequate  statements  of  educational  objectives 

13 

available  to  them."  J 

In  answer  to  this  need  various  attempts  at  classification  have 

Ik 

been  made.  Kearney  prepared  a  comprehensive  statement  of  elementary 
school  objectives  for  the  Mid-Century  Committee  on  Outcomes  in  Elementary 


12B.  G.  Massialas  and  A.  M.  Kazamias  (editors).  Crucial  Issues  in 
The  Teaching  of  Social  Studies  (Englewood  Cliffs,  New  Jersey:  Prentice- 

Hall  Inc.,  19^5-),  p.  219. 

12 

0.  L.  Davies,  Jr.  and  Drew  C.  Tinsley,  "Cognitive  Objectives 
Revealed  by  Classroom  Questions  Asked  by  Social  Studies  Teachers," 
(unpublished  paper  presented  to  the  1967  Annual  Meeting  of  The  American 
Research  Association). 

13Ebel,  op.  cite , 

14  . 

N.  C.  Kearney,  Elementary  School  Objectives  (New  York:  Russel, 

Sage  and  Co.,  1953)* 
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Education.  French  and  associates1^  prepared  somewhat  similar  treatments  of 
secondary  school  objectives.  The  Taxonomy  represented  the  most  compre¬ 
hensive  classification  to  date  in  the  cognitive  area  of  objectives.  In 
The  Handbook  I:  Cognitive  Domain  Bloom  outlines  why  these  objectives  were 
central  to  achievement  test  development. 

The  cognitive  domain  which  is  the  concern  of  this  handbook, 
includes  those  objectives  which  deal  with  the  recall  or 
recognition  of  knowledge  and  the  development  of  intellectual 
abilities  and  skills.  This  is  the  domain  which  is  most 
central  to  the  work  of  much  of  current  test  development.  It 
is  the  domain  in  which  most  of  the  current  work  in  curriculum 
development  has  taken  place  and  where  the  clearest  definitions 
of  objectives  are  to  be  phrased  as  descriptions  of  student 
behaviour . 


IV.  THE  VALIDATION  OF  OBJECTIVE  ATTAINMENT 


Once  a  set  of  objectives  is  arrived  at  there  is  the  need  to  con¬ 
struct  an  instrument  to  judge  its  attainment.  Also,  there  is  the  funda¬ 
mental  need  to  apply  the  tools  of  measurement  to  validate  its  use.  Dyer 
contended  that: 

At  the  present  time  there  is  a  tendency  to  think  carelessly 
of  achievement  in  any  academic  area  as  one  dimensional.  The 
single  final  grade  in  a  course,  the  single  score  on  most 
standardized  achievement  tests  reinforce  this  tendency.  Al¬ 
though  any  good  test,  as  noted  above,  represents  an  effort  to 
sample  several  aspects  of  learning  in  any  particular  field, 
there  is  almost  never  any  systematic  attempt  to  investigate 
the  hypotheses  inherent  in  the  test  construction  process.  The 
question  of  construct  validity  is  left  dangling.  ' 


W.  French  and  others,  Behavioural  Outcomes  of  General  Education 
in  High  School  (New  York:  Russel,  Sage  and  Co.,  19577 • 

Bloom,  0£.  cit . ,  p.  7* 

^H.  S.  Dyer,  "Educational  Measurement --Its  Nature  and  Problems," 
Evaluation  in  Social  Studies,  Thirty-Fifth  Yearbook  of  the  National 
Council  for  the  Social  Studies  (Washington:  NCSS,  1965).,  p.  ^1. 
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Several  attempts  have  been  reported  to  validate  testing  instruments 
based,  entirely  or  in  part,  on  The  Taxonomy. 

The  work  of  McGuire"1  involved  the  construction  and  use  of  683  test 
items.  Three  subject  matter  experts  independently  classified  these  items 
according  to  an  eight  level  taxonomy  of  intellectual  processes  adapted  from 
The  Taxonomy.  The  raters  unanimously  agreed  on  the  classification  of  6l 
per  cent  of  the  items  rated;  an  additional  13  per  cent,  one  rating 
diverged  from  the  other  two  by  only  one  level.  Two  or  more  raters  agreed 
on  93  per  cent  of  the  items.  An  attempt  to  determine  the  degree  of  cor¬ 
respondence  between  the  intellectual  processes  actually  employed  by 
examiners  and  that  described  by  the  experts  provided  results  of  limited 
usefulness. 

From  statistical  analysis  of  results  for  a  comprehensive  examina¬ 
tion  constructed  according  to  a  revised  taxonomy,  McGuire  concluded  that 
the  items  at  each  level  of  The  Taxonomy  appeared  to  be  measuring  somewhat 
different  intellectual  skills.  Findings  from  factor  analysis  of  the  cor¬ 
relation  matrices  supported  the  hypothesis  that  there  is  a  fundamental 
kind  of  knowledge,  understanding,  or  intellectual  skill  basic  to  all 
processes . 

19 

Stoker  and  Kropp  used  two  reading  comprehension  tests  in  which 
items  were  classified  according  to  the  six  levels  of  The  Taxonomy.  Class- 


C.  McGuire,  "Research  in  the  Process  Approach  to  the  Construction 
and  Analysis  of  Medical  Examination"  Twentieth  Yearbook  of  the  National 
Council  on  Measurement  in  Education,  (East  Lansing,  Michigan:  Office  of 
Evaluative  Services,  1963),  pp.  7  ~  16. 

^H.  W.  Stoker  and  R.  P.  Kropp,  "Measurement  of  Cognitive  Processes," 
The  Journal  of  Educational  Measurement  I:  39  “  42,  June,  1964. 
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ification  of  items  by  judges  and  the  application  of  Guttman's  simplex 
theory  to  inter-item  correlation  matrices  supported  the  hierarchical 
structure  of  The  Taxonomy.  There  was  evidence,  however,  that  the  evalua¬ 
tion  category  may  be  misplaced.  Factor  analysis  provided  evidence  to  sup¬ 
port  the  knowledge  -  higher  mental  process  division  of  items  but  failed  to 

differentiate  among  the  upper  levels  of  The  Taxonomy. 

20 

Connolly  and  Wantman  recorded  the  oral  responses  of  students  to 
test  items  based  on  Bloom's  Taxonomy ♦  Although  their  primary  motive  was 
to  identify  types  of  problem  solving  difficulties,  they  noted  a  division 
between  recall  or  memory  responses  and  those  exhibiting  the  use  of  higher 

mental  processes. 

21 

Berlin  in  doing  his  research  on  structured  learning  situations 
found  some  evidence  for  justifying  Bloom's  hierarchy  of  objectives.  In 
particular,  he  drew  attention  to  the  evident  division  between  responses 
to  the  knowledge  objective  and  those  above.  He  noted,  however,  that  the 
process  of  translation  which  was,  according  to  Bloom,  part  of  comprehen¬ 
sion  might  better  be  considered  a  knowledge  objective. 

22 

Ayers  conducted  extensive  factor  analysis  on  a  forty  item,  four 
option  multiple  choice  test  administered  to  297  students  in  an  Edmonton 

20 

J.  A.  Connolly  and  M.  J.  Wantman,  "An  Exploration  of  Oral  Reason¬ 
ing  Processes  in  Responding  to  Objective  Test  Items,”  Journal  of  Educa¬ 
tional  Measurement ,  I:  59-64,  June,  1964. 

21b.  M.  Berlin,  "The  Relationship  of  Learning  Experiences  of 
Students  to  Certain  Structured  Learning  Situation,”  (Doctoral  Thesis, 
University  of  Chicago,  1965)  (Microfilm) 

22 

J.  D.  Ayers,  "Justification  of  Bloom's  Taxonomy  by  Factor 
Analysis,"  (unpublished  paper,  Department  of  Educational  Psychology, 
University  of  Alberta). 
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School,  immediately  after  they  had  completed  a  Grade  10  science  unit  on  the 
gas  laws.  From  his  analysis  he  concluded:  "In  general  this  factor  ana¬ 
lytic  study  of  the  items  within  a  relatively  short  test  supports  Bloom's 
notion  of  hierarchical  structure,  but  there  is  some  question  whether  or 
not  the  same  factors  and  hierarchical  order  will  be  confirmed. "* 23 

Crawford  administered  a  collection  of  723  test  items,  divided  in¬ 
to  eight  item  groups  to  475  American  college  students  and  attempted  to 
ascertain  the  validity  of  these  items  in  testing  higher  mental  processes. 
Rotated  factors  were  extracted  from  forty-two  subject  matter  and  thirty- 
nine  taxonomic  matrices.  The  factors  so  extracted  could  not  be  defined 
in  terms  of  levels  of  mental  processes.  "The  patterns  of  factors  obtained 
suggested  that  items  from  the  same  subject  matter  which  load  on  different 
factors  may  define  subject  matter  sub -categories,  and  that  items  from  dif¬ 
ferent  subject  matter  categories  which  have  loadings  on  the  same  factor 

25 

might  deal  with  the  same  general  topic  or  concept."  y 

Taking  the  concurrent  approach  to  validation,  McFall  constructed 
and  conducted  a  study  of  an  experimental  test  composed  of  35  multiple 
choice  items  dealing  with  the  area  of  general  science  and  concentrating 

23Ibid.,  p.  4. 

2i*William  R.  Crawford,  "The  Empirical  Validity  of  Items  Constructed 
to  Measure  Complex  Intellectual  Processes,"  (an  unpublished  paper  pre¬ 
sented  to  the  1967  Annual  Meeting  of  The  American  Educational  Research 
Association) . 

25Ibid.,  p.  7. 

2^Robert  W.  McFall,  "The  Development  and  Validation  of  an  Achieve¬ 
ment  Test  for  Measuring  Higher  Level  Cognitive  Processes  in  General 
Science,"  The  Journal  of  Experimental  Education,  33:  103-6,  Fall,  1964. 
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specifically  on  the  relationship  between  plants  and  animals.  The  first 
section  of  the  test  (subtest  A)  included  items  designed  to  provide  an  in¬ 
dication  of  the  student's  ability  to  recall  specific  and  isolated  bits  of 
information.  The  second  section  (subtest  B)  utilized  a  problem  solving 
approach  in  order  to  identify  the  ability  to  deal  with  tasks  requiring 
higher  level  cognitive  processes.  Teachers  administered  the  test  to  17 
general  science  classes  in  grades  seven  to  eleven.  Subtest  results  were 
correlated  with  three  "current  methods  of  evaluating  student  achievement" 
as  follows  % 

(a)  present  class  grade 

(b)  total  score  The  Stanford  Achievement  Test  J  -  advanced 
battery  -  1953 

(c)  score  on  the  Science  Section  of  the  Stanford  Achievement 
Test  (Form  J  -  advanced  battery  -  1953) 

McFall's  hypotheses  were  supported  as  in  all  three  comparisons  sub¬ 
test  A  had  a  significantly  higher  correlation  with  the  external  criteria 
than  did  subtest  B.  He  found  a  similar  relationship  between  the  subtests 
and  scores  on  The  Otis  Quick-scoring  Mental  Abilities  Test.  In  his  con¬ 
cluding  statement  he  wrote,  "a  well  validated  measuring  device  constructed 
to  reflect  the  need  to  emphasize  critical  understandings  and  problem¬ 
solving  skills  in  conjunction  with  the  acquisition  of  content  material  can 

facilitate  a  review  of  educational  goals  and  the  processes  directed  toward 

27 

the  realization  of  these  objectives." 


2^Ibid. ,  p.  106. 
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V.  THE  MEMORY  FACTOR 

The  attempt  to  isolate  a  basic  memory  or  knowledge  objective  has  a 

history  which  goes  beyond  Bloom ' s  Taxonomy . 

28 

Cronbach  said  that  the  depth  of  understanding  depends  in  part  on 
the  nature  of  the  material,  and  that  its  measurement  depends  on  the  kinds 
of  items  used.,  There  is  a  suggestion  in  discussions  such  as  this  that  re¬ 
call  of  knowledge  depends  on  memory  alone  and  that  reasoning  involved  more 
than  this. 

Thurstone2^  identified  a  memory  factor  in  his  factor  analytic 
studies  and  included  it  among  his  "Primary  Mental  Abilities.”  The  tests 
which  were  related  to  this  factor  involved  paired  associates  and  recogni¬ 
tion  items. 

Hilgard^  cited  experiments  which  showed  that  memory  for  number 
sequences  was  better  when  a  principle  of  organization  was  understood, 
than  when  the  sequences  were  learned  by  rote.  Another  experiment  indi¬ 
cated  that  memory  of  application  of  principles  in  college  biology  was 

better  than  memory  for  terminology. 

•21 

More  recently  Vernon-^  has  reviewed  the  evidence  for  memory 

28 

L.  J.  Cronbach,  Ecuational  Psychology  (New  York:  Hare ourt , 

Brace  and  Company,  1954). 

29l.  L.  Thurstone,  Primary  Mental  Abilities  (Chicago:  The 
University  of  Chicago  Press,  19 3o) • 

3(-)E.  R.  Hilgard,  Introduction  to  Psychology  (New  York:  Harcourt, 
Brace  and  Co.,  1957)* 

^^P.  E.  Vernon,  The  Structure  of  Human  Abilities ,  (London: 

Metheuen  and  Co.  Ltd.,  1961). 
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factors.  In  doing  so  he  pointed  out  that  Thurstone's  tests  do  not  bear 
much  resemblance  to  learning,  retention  or  recall  in  everyday  life.  He 
cited  other  evidence  to  support  che  existence  of  a  rote  memory  factor,  but 
noted  that  in  all  cases  short  term  memory  is  involved.  In  an  investigation 
where  delayed  memory  was  tested,  the  tests  did  not  load  on  the  rote  memory 
factor  but  on  the  verbal  factor. 

HukinsJ^  in  his  factorial  investigation  of  science  tests,  after 
reviewing  evidence  for  a  recall  factor,  concluded  that: 

In  a  factorial  investigation,  a  test  of  knolwedge  in¬ 
volving  simple  recall  type  items  would  have  high  loadings 
on  a  verbal  factor  if  such  a  factor  is  obtained  in  the 
analysis.  If  a  general  intellectual  ability  factor  is 
obtained  the  test  of  knowledge  probably  would  have  loading 
on  this  factor  also.  However,  if  the  analysis  yields  a 
reasoning  and  a  verbal  factor,  it  is  likely  that  the  test  ^ 
would  have  small  or  zero  loading  on  the  reasoning  factor." 


VI.  SUMMARY 


Writers  have  expressed  the  need  to  provide  achievement  tests  which 
are  more  consistent  with  objectives.  Also  described  is  the  tendency  for 
these  evaluative  instruments,  as  they  now  exist,  to  overemphasize  the 
knowledge  objective. 

Proposed  solutions  to  the  problems  have  had  two  stages.  First, 
there  have  been  efforts  to  provide  clearer  and  more  complete  classifica¬ 
tions  of  objectives.  Second,  there  has  been  the  effort  to  construct  and 

3^A.  A.  Hukins,  "A  Factorial  Investigation  of  Achievement  of 
Objectives  in  Science  Teaching"  (unpublished  Ph.  D.  thesis.  Department  of 
Secondary  Education,  University  of  Alberta,  1963),  p.  10. 

33Ibid.,  p.  10. 
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validate  objective  tests  which  evaluate  these  objectives.  In  this  second 
stage  subject  area  evaluators  have  made  much  use  of  the  research  base  of 
educational  psychologists  who  have  attempted  to  identify  and  test  for 
memory  and  other  mental  abilities. 

Bloom’s  Taxonomy  is  one  product  of  the  work  of  educational  psycholo¬ 
gists  which  has  and  is  finding  considerable  application  in  various  subject 
fields,  particularly  science.  Research  to  date  has  been  inconclusive  in 
validating  the  entire  hierarchy  of  the  Taxonomy  in  different  subject 
areas  but  studies  have  shown  fairly  consistent  identification  of  higher 
mental  processes  as  distinct  from  knowledge  or  recall. 


. 


CHAPTER  III 


DESIGN  OF  THE  INVESTIGATION 

I.  THE  EXAMINATION  -  PART  I 

The  1966  Grade  IX  Social  Studies  Departmental  Examination  was  the 
product  of  a  six  member  committee  working  under  the  auspices  of  the  High 
School  Entrance  Examinations  Board.  The  multiple  choice  items  initially 
chosen  from  a  bank  of  approximately  five  hundred  items  for  Part  I  of  the 
examination  were  used  in  a  pilot  study  in  which  they  were  administered  to 
a  group  of  Edmonton  Grade  X  Students.  The  results  of  this  study  were 
used  as  a  basis  for  accepting,  rejecting  and  modifying  questions  to  at¬ 
tempt  to  insure  that  the  items  appearing  on  the  final  version  of  the  test 
evinced  reliability  and  discriminatory  power.  One  hundred  multiple  choice 
items,  thus  arrived  at,  were  administered  to  Alberta  Grade  IX  Students  as 
Part  I  of  the  examination. 

The  biserial  correlation  with  the  total  test  score  and  the  diffi¬ 
culty  Index  for  each  item  is  recorded  in  Appendix  A. 

II.  THE  SAMPLE 

The  sample  used  in  this  investigation  consisted  of  958  sets  of 

examination  scores  drawn  from  a  total  student  population  of  26,111 

writing  the  examination.  Random  selection  was  insured  through  the  use  of 

1 

Fisher  and  Yates  table  of  random  numbers.  As  Stuart  has  stated  this 


Ronald  A.  Fisher  and  Frank  Yates,  Statistical  Tables .  (London: 
Oliver  and  Boyd,  1948),  pp.  104-9. 
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procedure  "is  not  the  abdication  of  responsibility  but  the  very  reverse: 

we  insure  that  no  human  inclination  or  prejudice  will  be  allowed  to  en- 

2 

danger  the  bias  free  nature  of  the  sample." 

Each  of  the  958  sets  of  scores  consisted  of  the  following  four 

items : 

(a)  the  record  of  responses  to  the  objective  questions  of  Part  I 
of  the  examination, 

(b)  the  principal's  rating  for  Social  Studies  IX, 

(c)  the  total  score  and  numerical  and  verbal  sub-scores  obtained 
on  the  School  and  College  Ability  Test, 

(d)  the  score  obtained  on  Part  II,  the  subjectively  scored 
section  of  the  examination. 

The  size  of  the  sample  was  to  some  extent  governed  by  considera¬ 
tions  external  to  the  study.  The  Examinations  Branch  wished  to  utilize 
the  sample  for  statistical  work  and  in  the  past  had  used  a  sample  of  this 
size.  Although  the  size  of  the  sample  exceeded  the  minimum  requirements 
of  the  statistical  tests  used,  using  the  sample  in  its  entirety  was 
thought  advisable.  "The  basic  justification  of  statistical  inference  is 
that  the  distribution  which  is  obtained  from  a  random  sample  tends  to 

resemble  the  population  from  which  it  is  drawn.  This  tendency  increases 

3 

as  the  size  of  the  sample  increases." 

^A.  Stuart,  Basic  Ideas  of  Scientific  Sampling .  (London:  Charles 
Giffen  and  Company,  Ltd . ,  1962 ) ,  p.  11. 

^Helen  M.  Walker  and  Joseph  Lev,  Elementary  Statistical  Methods ♦ 
(New  York:  Holt,  Rinehart,  and  Winston,  1958)>  p*  208. 
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III.  CONSTRUCT  VALIDITY 

To  determine  whether  evidence  of  a  division  or  divisions  existed 
among  recall  and  higher  mental  process  items  or  sets  of  items  of  the 
examination,  tests  of  construct  validity  based  on  the  hypotheses  and 
assumptions  of  Chapter  I  were  applied  to  the  sample. 

Identification  of  Items  by  Judges 

Twelve  judges,  each  working  independently,  identified  each  of 
the  multiple  choice  items  according  to  the  six  major  categories  of  The 
Taxonomy.  Each  person  chosen  satisfied  two  criteria:  (l)  each  was 
familiar  with  The  Taxonomy  and  (2)  each  had  recently  taught  or  was 
currently  engaged  in  teaching  the  Social  Studies  IX  course. 

These  identifications  were  recorded.  After  each  person  had 
completed  the  identification  of  test  items,  he  was  interviewed  as  to  the 
reason  for  his  selections. 

The  proportion  of  agreement  among  the  judges  was  summarized  and 
then  a  comparison  between  the  work  of  the  judges  and  the  initial  item 
identification  of  the  committee  constructing  the  examination  was  made. 

Factor  Analysis 

In  assessing  construct  validity,  the  test  investigator  wishes  to 
infer  the  degree  to  which  the  individual  possesses  some  hypothetical 
trait  or  construct  presumed  to  be  reflected  in  the  test  performance.  In 
this  investigation,  these  are  the  subjectively  identified  ability  levels 
of  The  Taxonomy. 

Factor  analysis  was  chosen  as  the  statistical  technique  suited  to 
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assessing  the  construct  validity  of  the  test.  Guilford  maintained  that 

"the  best  answer  to  the  question  'What  does  this  test  measure?'  is  in  the 

form  of  a  list  of  primary  factors  with  which  it  correlates  and  their  pro- 

4 

portion  of  variance  in  the  test."  The  above  validity  estimate  is  known 
as  factorial  validity.  According  to  Guilford  this  type  of  validity  is 
basic  to  the  understanding  of  other  kinds  of  validity  and  of  many  phenomena 
of  correlation  in  general. 

The  Theory.  Factor  analysis  assumes  that,  if  two  tests  are  admini¬ 
stered  to  a  group  of  individuals  and  the  resulting  scores  indicate  that 
there  is  a  high  correlation  between  the  tests,  the  tests  are  measuring 
some  underlying  common  factor  or  hypothetical  variable.  Positive  correla¬ 
tions  between  a  number  of  tests  or  test  items  indicate  that  one  or  more 
common  factors  are  operating  in  the  domain.  Factor  analysis  provides  a 
means  for  finding  out  information  about  these  hypothetical  variables. 

The  variance  of  a  set  of  obtained  scores  for  a  test  is  composed 
of  true  variance,  arising  from  individual  differences  of  the  subjects  and 
error  variance  resulting  from  errors  of  measurement.  When  there  is  cor¬ 
relation  between  a  number  of  measures  some  of  the  true  variance  will  be 
common  to  two  or  more  lests,  and  some  of  it  will  be  specific  to  each.  In 
factor  analysis,  it  is  the  common  variance  which  is  of  special  interest. 

If  a  number  of  hypothetical  variables  less  in  number  than  the  tests  (in 
this  case  the  test  items  and  subtests)  can  be  found  to  express  the  common 
variance  and  thus  the  common  part  of  the  test  scores,  then  parsimony  has 


J.  P.  Guilford,  Fundamental  Statistics  in  Psychology  and  Educa¬ 
tion.  (4th  ed.)  New  York:  McGraw-Hill,  1965 ,  p»  472. 
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been  achieved.  If,  in  addition,  meaning  can  be  given  to  these  common  fact¬ 
ors,  then  an  ordering  of  thinking  about  the  domain  under  investigation  can 
be  achieved. 

Ma thematically,  factor  analysis  is  based  on  a  linear  model.  The 
score  Z  for  an  individual  on  a  test  is  represented  as  a  linear  combina¬ 
tion  of  hypothetical  variables -- 

Z..  =  a./F/  +  a.2F2  +  a  F  +  a  U  : 

J  i  j  i  jiu  mi  .i  ii 


J  ji 


--where  Z._^  is  the  score  of  person  i  on  test  j  in  standard  form,  (with 
mean  0,  and  standard  deviation  l)  a^  (p  =■  1,  2....m)  is  the  factor  load¬ 
ing,  and  expresses  the  correlation  between  tests  j  and  factor  p,  F  is  the 

Pi 

factor  score  and  describes  the  individual  i  in  relation  to  factor  p. 

U  (i  =  1,  2....N)  (j  =1,  2...  .in)  are  the  unique  factors;  they  describe 
Ji 

the  person  i  in  relation  to  the  unique  part  of  j,  a,  (j  =  1,  2....n)  is  a 

J 

factor  loading  expressing  the  correlation  between  tesi  j  and  the  unique 
factor  corresponding  to  test  j . 

For  a  given  person  there  are  n  values  of  z  ,  one  for  each  of  n 

ji 

tests,  and  there  are  N  values  of  i,  one  for  each  of  N  individuals. 

In  order  to  gain  information  about  a  group  of  tests  being  analyzed, 

the  problem  consists  of  finding  the  values  of  the  coefficients  a  for  the 

JP 

common  factors . 

To  derive  loadings  or  coefficients  on  factors  in  as  many  meaning¬ 
ful  ways  as  possible  in  terms  of  The  Taxonomy's  process  levels  two  ap¬ 
proaches  were  used  in  grouping  the  test  results  for  analysis . 


Individual  Item  Analysis .  The  first  approach  involved  factor 
analysis  based  on  individual  multiple  choice  items  and  somewhat  paralleled 
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the  methods  with  which  Ayers  experienced  some  success. 

Tetrachoric  correlations  were  calculated  for  the  mu  tiple  choice 
items  of  Part  I. 

Tetrachoric  correlation  is  appropriate  to  data  arranged  in 
a  2  x  2  or  fourfold  table.  It  assumes  that  both  variables 
underlying  the  dichotomies  are  normally  distributed.  It  has 
b~en  used  to  provide  a  convenient  measure  of  correlation  when 
graduated  measurements  have  been  reduced  to  two  categories. 

It  is  an  estimate  of  pro duct -moment  correlation. 

The  standard  error  or  r^  is  always  greater  than  the 
standard  error  of  r.  The  magnitude  of  the  error  increases  with 
the  extremity  of  the  dichotomies. 

6 

The  procedure  is  quite  acceptable  when  N  is  large. 

Factors  from  five  to  twenty-five  in  number  were  extracted  from  the 
correlation  matrix  thus  derived,  and  rotated  in  order  to  find  the  most 
meaningful  factorial  arrangement. 


Subtest  Analysis .  The  second  approach  involved  factor  analysis 
based  on  item  subtests  and  somewhat  paralleled  the  methods  with  which 

7  8 

Stoker  and  Kropp  and  Crawford  did  their  research. 

Items  were  divided  into  seven  subtests  on  the  basis  of  units  of 
content.  Then  they  were  subdivided  into  knowledge  item  and  higher  mental 
process  item  subtests,  resulting  in  the  thirteen  subtests  as  shown  in 
Figure  II. 

CorTelations  were  calculated  for  these  subtests  and  from  three  to 


Ayers ,  0£.  cit. 

Guilford,  0£.  cit.  ,  p.  2kk~6. 
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five  factors  were  extracted  and  rotated  in  order  to  find  the  most  meaning 
ful  factorial  arrangement  by  subtests. 

IV.  CONCURRENT  VALIDITY 

In  order  to  investigate  the  concurrent  validity  of  the  test,  the 
items  of  Part  I  were  divided  into  two  twc-subtest  groupings.  Subtest  1A. 
was  composed  of  knowledge  or  recall  items  and  subtest  IB  was  composed  of 
questions  designed  to  test  the  higher  mental  process.  The  classification 
reflected  in  subtests  1A.  and  IB  were  those  of  the  committee  which  con¬ 
structed  the  examination.  Subtests  2A  and  2B  were  similarly  composed  of 
knowledge  and  higher  mental  process  questions  with  the  classification  here 
being  that  of  the  majority  of  the  group  of  judges.  These  subtests  were 
correlated  with  certain  selected  criteria. 

External  Criteria 

Principal’s  ratings..  Scholastic  and  College  Ability  Test  Scores, 
and  sub-scores  from  Part  II,  the  sub  jective^  essay  section  of  the  examina¬ 
tion  were  chosen  as  external  criteria.  Each  set  of  scores  represented  a 
distinct  form  of  evaluation  for  grade  IX  students.  All  scores  were  ob¬ 
tained  at  the  same  time  and  under  the  same  conditions  as  the  subtests  of 
Part  I,  previously  mentioned. 

Principal *  s  Ratings .  ’’These  marks  are  often  based  on  a  combination 

of  the  teacher’s  estimate  of  student  intelligence,  maturation  and  class- 

q 

room  performance  during  the  year."'  In  many  cases  these  marks  are  also 

9 

Donald  B.  Black  and  Donald  W.  Knowles.  "The  Effectiveness  of 
The  Grade  XII  Principal's  Rating  Scores  to  predict  Freshman  Success  in 
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influenced  by  the  pupil’s  performance  on  final  examination  of  the  previous 

n  ^  10 

year.  Black’s  study  of  1958  revealed  a  .65  correlation  between  the 
Departmental  Mark  in  Social  Studies  IX  and  the  Principals’  Eatings.  These 
ratings  represent  a  teacher’s  professional  assessment  of  pupil  performance 
and  are  thought  to  have  important  implications  for  prediction  and  for  the 
eventual  accreditation  of  schools. 

Co-operative  School  and  College  Ability  Tests.  Published  by  the 
Educational  Testing  Service,  this  test  reflects  a  shift  away  from  measur- 
ing  "mental  ability"  to  a  greater  concern  with  school  learned  abilities 
when  one’s  aim  is  to  predict  school  success.  According  to  the  manual 
the  main  purpose  of  the  test  is  "to  estimate  the  capacity  of  each  indivi¬ 
dual  student  to  undertake  the  academic  work  of  the  next  higher  level  of 
11 

schooling.  This  test  is  divided  into  four  parts.  Parts  I  and  III  are 

concerned  with  "developed  ability  in  skills  that  are  closely  related  to 

,,12 

student  success  in  the  verbal  kinds  of  school  learning.  Parts  II  and 
IV  are  "measures  of  ability  in  certain  quantitative  skills  of  number 

9  (cont'd  ) 

Four  Faculties  of  The  University  of  Alberta,"  The  Alberta  J ournal  of 
Educational  Research  XI:  128  No.  2. 

■^Bonald  B.  Black  and  Donald  W.  Knowles.  "A  Study  of  The  Relation¬ 
ship  of  The  Grade  IX  Principal’s  Rating  to  Performance  on  the  Alberta 
Grade  IX  Departmental  Examinations,"  The  Alberta  Journal  of  Educational 
Research  XTV:  227-238,  1958. 

11 

School  and  College  Ability  Tests,  School  Ability  Tests,  Manual 
for  Interpreting  Scores.  ^Princeton,  New  Jersey:  Educational  Testing 
Service,  1955) > P •  3 • 

^Ibid. 
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13 

manipulation  and  problem  solving."  Thus  the  test  gives  a  verbal  score 
and  quantitative  score  and  these  may  be  combined  to  give  a  total  score,  if 
desired.  The  items  are  all  of  the  five  response  multiple -choice  type. 

The  reviewers  in  The  Fifth  Mental  Measurement  Yearbook  were  general  ■ 
ly  agreed  concerning  the  high  quality  of  the  construction  of  these  tests. 

The  reliability  estimates  given  by  the  manual  are  .93  for  the  Verbal  Score 

Inl¬ 
and  .91  for  the  Quantitative  Score.  The  publishers  also  gave  evidence 

of  the  content  and  concurrent  validity  of  the  test  in  the  manual  and  evi- 

dence  of  its  predictive  validity  in  a  later  supplement. 

Part  II  Sub -Scores .  Part  II  of  the  examination  was  of  the  sentence 
answer  and  short  essay  type.  The  sub-score  obtained  on  this  part  of  the 
examination  comprised  thirty  per  cent  of  the  total  mark.  This  sub-score 
was  assigned  by  teachers  marking  the  examination  subjectively  under  the 
auspices  of  The  Examination  Branch  of  The  Alberta  Department  of  Education. 

Inter c orre lat ion 

The  subtests  of  Part  I  were  correlated  with  the  external  criteria 
using  Pearson's  product -moment  correlation.  Use  of  the  Pearson  r  is 
governed  by  a  number  of  basic  assumptions  which  could  be  met.  The  data 
used  was  scored  on  interval  scales ,  could  be  expected  to  show  normality  of 
arrays ,  and  the  relationship  of  one  set  of  scores  to  each  other  was 
rectilinear. 

Other  methods  of  correlation  were  considered  hut  it  was  felt  that 

13 

Ibid. 
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the  Pearson  r  would  provide  the  most  accurate  indication  of  the  relation¬ 
ship  being  calculated. 

To  test  Hypotheses  B1  and  B2  a  test  of  significance  was  applied  us¬ 
ing  the  distribution  of  t  where  the  t  value  required  is  given  by  the 


N~2 


formulas  t 


*  r 


To  test  Hypothesis  B3  the  significance  of  the  difference  in  the  cor¬ 
relation  of  item  subtests  with  external  criteria  was  calculated  using  the 


formulas. 


(r12  “  r13^  +  3)  (1  +  r23) 


Since  this  study  proposed  to  investigate  and  report  and  recommend 
for  further  study  rather  than  be  a  basis  for  direct  action  all  levels  of 
significance  were  reported  in  the  results. 

To  decide  significance  in  correlations  and  differences  in  correla¬ 
tions  an  alpha  level  of  .05  was  set.  Use  of  this  significance  level  was 
justified  on  the  following  grounds.  Choosing  an^  of  .05  enlarges  the 
possibility  of  T/pe  II  error j  of  accepting  Hq  when  in  fact  it  is  falser 
as  opposed  to  using  an  of  .01  which  enlarges  the  possibility  of  Type  I 
error f  rejecting  Hq  when  in  fact  it  is  true.  Choosing  .05  enlarges  the 
chance  of  finding  significance  in  this  situation.  As  Seigel  points  out., 
"the  level  at  which  the  researcher  chooses  to  set  alpha  should  be  de¬ 


termined  by  his  estimate  of  the  importance  of  the  practical  significance 


■ 
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of  his  findings . This  study  will  reflect  on  the  possibility  of  further 
validating  such  tests  and  will  not  lead  directly  to  action,  thus,  it  was 
felt  the  alpha  level  of  .05  would  serve. 

V.  SUMMARY 

All  major  statistical  calculations  for  the  study  were  made  through 
use  of  programming  and  computer  facilities  available  through  the  Faculty 
of  Educations  Research  Services. 


15 

Sidney  Seigel,  Nonparametric  Statistics  for  The  Behavioural 
Sciences  (New  York;  McGraw-Hill  Book  Company,  195^)>  p.  9* 
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CHAPTER  IV 


ANALYSIS  AND  RESULTS  RELATED  TO  HYPOTHESES 

I.  CONSTRUCT  VALIDITY 


Hypothesis  Al 

A  group  of  twelve  judges ,  judging  independently  will 
reach  agreement  in  classifying  objective  items  of  the  test 
into  knowledge  and  higher  mental  process  categories. 

Agreement  Among  Judges :  Twelve  judges  reached  unanimous  agreement 
in  identifying  thirty-four  test  items ,  agreement  with  only  one  dissenting 
placement  on  fifteen  items  and  majority  (7  or  more)  agreement  on  an  addi¬ 
tional  forty  items.  This  made  a  total  of  eighty -nine  items  on  which 
majority  or  better  agreement  was  reached  in  identifying  items  according  to 
the  six  major  levels  of  The  Taxonomy.  For  nine  of  the  eleven  items  on 
which  there  was  less  than  majority  agreement ,  the  judges  showed  disagree¬ 
ment  only  with  respect  to  which  of  the  higher  mental  processes  the  questions 
required.  Hypothesis  A1  was  supported  in  that  on  only  two  of  the  one  hun¬ 
dred  items  did  the  judges  show  less  than  majority  agreement  in  placing  an 
item  in  the  knowledge  or  recall  category  as  opposed  to  one  of  the  higher 
mental  process  categories. 

Agreement  Between  Judges  and  Committee ;  On  the  eighty-nine  items 
on  which  the  judges  expressed  majority  or  better  agreement,  sixty-four  of 
these  identifications  agreed  with  the  initial  placement  by  the  committee 
which  designed  the  examination.  For  twenty -three  of  the  twenty-five  items 
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on  which  a  majority  of  the  judges  disagreed  with  the  initial  placement,  the 
judges  placed  the  item  at  a  lower  process  level  than  that  set  by  the 
committee . 

Eleven  of  the  items  had  been  given  dual  identifications  by  the 
committee;  that  is,  the  committee  anticipated  that  students  might  answer 
the  questions  using  one  of  two  possible  mental  processes.  For  seven  of 
these  items  a  majority  of  the  judges  chose  the  lower  of  the  two  categories 
in  terms  of  The  Taxonomy.  For  the  remaining  four  items  a  majority  of  the 
judges  chose  a  level  or  category  below  the  lower  of  the  two  alternatives 
given  by  the  committee . 

When  the  individual  judges  were  interviewed  with  respect  to  their 
lack  of  agreement  with  the  initial  classification,  two  comments  were 
recurrent.  First,  judges  agreed  that  certain  items  could  elicit  responses 
at  a  higher  level  of  thought  than  they  had  categorized  the  items  as  doing, 
but  when  they  considered  their  knowledge  of  classroom  situations  and  the 
manner  in  which  students  are  taught  they  thought  it  unlikely.  Second, 
they  noted  that  in  many  cases,  students  could  have  used  a  process  of 
eliminating  dj  ^tractors  and  arrived  at  the  correct  answer  without  having 
“thought  through"  the  question  in  the  manner  intended. 

The  a  priori  classification  of  the  judges  for  the  items  of  Part  I 
and  the  complete  set  of  responses  by  the  judges  are  recorded  in  Appendix 
C. 


Hypotheses  A2  and  A3 

(A2)  The  responses  to  the  objective  items  classified 
in  each  Taxonomy  category  will  correlate  positively. 
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,.A3)  Factor  analysis  of  correlation  among  items  will 

result  in  higher  mental  process  items  loading  on  factors  in  a 

manner  distinct  from  the  loading  of  recall  items. 

As  outlined  in  Chapter  III  factor  analysis  was  used  to  examine  the 

relationships  among  test  items  and  subtests.  "One  method  for  checking  out 

hypotheses  of  this  sort  is  known  as  factor  analysis.  With  the  availability 

of  modern  computers ,  it  provides  a  powerful  and  efficient  means  of  test- 

1 

ing  out  constructs  that  have  presumably  been  built  into  tests."  The  fol¬ 
lowing  is  an  explanation  of  the  procedure  of  rotating  factors  used  to 
arrive  at  the  results  of  this  part  of  the  study  and  a  description  of  the 
means  of  analysis  used  in  arriving  at  the  results  that  are  reported. 

Rotation  of  Factors ♦  In  factor  analysis  each  test  is  represented 
by  a  point  in  a  common  factor  space  or  by  a  line  joining  the  origin  to 
the  point  and  called  a  test  vector.  The  correlations  between  tests  are 
determined  by  the  angles  between  the  test  vectors,  and  the  loadings  of  the 
tests  on  the  factors  are  given  by  the  perpendicular  projections  of  the 
test  vectors  on  the  reference  axis.  So  long  as  the  test  vectors  are 
fixed,  the  relationships  between  the  tests  are  not  altered  by  rotating  the 
reference  axis  about  the  origin  but  for  every  position  of  these  axes  there 
will  be  a  different  set  of  factor  loadings.  It  can  be  seen  that  there 
will  be  an  infinite  number  of  solutions  corresponding  to  all  possible 
placements  of  the  reference  axis .  The  particular  method  of  analysis 
places  the  axis  in  a  certain  position  which  may  or  may  not  have  special 

1  . 

Dyer,  0£.  cit . ,  p.  40. 
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significance  in  terms  of  psychological  meaning.  The  task  of  the  investi- 
gator  is  then  to  rotate  the  reference  axis  in  search  of  a  solution  which 
is  meaningful  in  terms  of  the  constructs  being  studied.  In  this  study  it 

was  a  case  of  arriving  at  a  factorial  arrangement  meaningful  in  terms  of  a 

% 

twofold  division  of  constructs  based  in  The  Taxonomy. 

Analysis s  Factor  analysis  for  both  item  and  subtest  arrangements 
consisted  of  utilizing  a  principal  components  method  of  analysis  in  which 
the  factors  are  represented  by  the  principal  axes  of  ellipsoids  formed 
through  the  plotting  of  scores  on  individual  variables.  The  first  princi¬ 
pal  factor  accounts  for  the  maximum  possible  variance  and  each  successive 
one  accounts  for  a  maximum  of  the  then  existing  residual  variance. 

From  the  principal  factor  solution  three  orthogonal  rotations— 

p 

equamax,  quartimax,  and  varimax— based  on  the  criteria  of  Harman  were 
performed.  Because  the  factor  pattern  derived  from  rotation  is  dependent 
upon  the  number  of  factors  entering  the  rotation,  it  was  decided  to 
iterate  the  rotations  using  five,  eight,  ten,  and  twenty -five  factors  for 
inter-item  analysis  and  three  and  five  factors  for  subtest  analysis. 

To  further  explore  for  the  most  meaningful  arrangement  of  items  and 
subtests  an  oblique  rotational  system,  promax,  was  applied  to  the  twenty- 
five  factor  varimax  solution  for  items  and  the  five -factor  equamax  solu¬ 
tion  for  subtests.  The  promax  solution  used  was  based  on  the  design  of 

3 

Henrickson  and  White.  Basically  it  is  a  procedure  for  constructing  the 


^H.  H.  Harman.  Modern  Factor  Analysis  (Chicago:  University  of 
Chicago  Press,  i960).  Ch.  l4. 

3 

A.  E.  Henrickson  and  P.  0.  White.  "Promax,  A  Quick  Method  for 
Rotation  to  Oblique  Simple  Structure,"  British  Journal  of  Statistical 
Psychology,  1964,  17,  pp.  65-70. 
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"best  possible"  oblique  solution  from  an  orthogonal  rotation.  It  is  de¬ 
signed  to  make  the  relationship  in  an  orthogonal  factor  matrix  clearer  by 
increasing  higher  loadings  on  factors  while  decreasing  smaller  loadings. 

Results  of  Inter-Item  Factor  Analysis;  Hypotheses  A2  and  A3  were 
not  supported  by  the  results  of  factor  analysis  in  that  no  clear  inter¬ 
pretation  could  be  made  in  terms  of  recall  and  higher  mental  process 
factor  divisions. 

Throughout  the  analysis  a  large  number  of  the  items  loaded  consist¬ 
ently  high  (.3  level  and  above)  on  one  seemingly  general  factor.  The  per¬ 
centage  of  the  total  variance  accounted  for  by  this  factor  was  calculated 
to  be  forty  per  cent  on  the  five  factor  principal  factor  solution.  A 
large  proportion  of  both  the  knowledge  or  recall  and  higher  mental  process 
items  loaded  on  this  first  factor . 

Examination  of  the  remaining  factors  in  the  factorial  arrange¬ 
ments  did  not  produce  any  clear  division  of  items  in  terms  of  a  process 
dimension.  Neither  did  the  results  produce  clear  content  divisions  among 
the  items  which  could  have  been  a  basis  for  second-order  factoring. 

However^  examination  of  the  results  obtained  did  show  certain  pat¬ 
terns  in  item  loadings  which  were  meaningful  as  a  basis  for  discussing 
the  inconclusiveness  of  the  results  and  pointing  out  avenues  for  further 
research. 

A  number  of  items  which  were  categorized  as  requiring  the  same 
mental  process  and  had  an  outline  somewhat  independent  of  course  content 
loaded  heavily  on  the  same  factor.  Test  items  64  to  67  and  Table  1, 
taken  from  the  promax  solution  shown  in  Appendix  D_,  illustrate  this  pat- 
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tern*  Items  which  were  structured  around  a  paragraph  or  some  similar  base 
tended  to  load  together  on  certain  factors. 

Head  the  following  paragraph  taken  from  Canada  and  The 
Western  World  before  answering  items  64  to  67. 

"The  BNA  Act  is,  in  effect,  the  written  part  of  the 
constitution  of  Canada,  but  there  is  a  vast  part  of  our 
constitution  that  is  unwritten  and  has  just  developed 
through  custom.  One  example  of  this  is  the  way  in  which 
bills  are  introduced  into  parliament  and  what  happens 
to  them  before  they  become  law.” 

Examine  each  statement  below  and  on  the  answer  sheet  mark: 

A.  if  the  statement  involves  the  BNA  Act. 

B.  if  the  statement  involves  parliamentary  procedure  or 
the  unwritten  consitution  of  Canada. 

C.  if  the  statement  involves  neither  of  the  above. 

John  Adams,  an  opposition  member  from  White  County  , 

P.  E.  I. ,  introduced  a  bill  into  the  House  of  Commons  stat¬ 
ing  that  a  program  for  research  that  will  lead  to  potato 
seed  improvement  be  sponsored  by  the  government  of  Canada. 

64.  A  government  member  from  Saskatchewan  claims  Adams 
is  out  of  order. 

65.  A  member  from  Ontario  thinks  that  it  is  a  good  idea. 

66.  A  member  from  B.  C.  claims  this  is  a  matter  for 
P.  E.  I.'s  provincial  government. 

A  number  of  other  items  which  dealt  with  distinctly  different  areas 


' 


■ 


45 


TABLE  I 

SELECTED  FACTOR  LOADINGS  FOR 
ITEMS  SIXTY -FOUR  TO  SIXTY-SEVEN 


Item 

Number 

Item 

Category 

Factor  Loadings 

Above  . 3  Level 

- 

(5) 

64 

3.00-4.00 

6838 

65 

3.00-4.00 

6962 

66  - 

3.00-4.00 

6834 

67 

3.00-4.00 

6058 

. '  '  '  di 


of  content  but  were  categorized  as  requiring  the  same  thought  process 
loaded  together  on  the  same  factors.  Items  35  an d  36  and  Table  II 
illustrate  this  pattern. 

Under  the  provision  of  "The  Alberta  Labour  Act"  a  Conciliation 
Commissioner  vas  appointed  to  settle  a  dispute  between  an  employer 
and  a  labour  union  over  wages.  A  Conciliation  Board  was  later 
appointed.  The  Conciliation  Board's  award  led  to  a  lockout. 

35*  It  can  be  stated  that  the  lock-out  was 

A.  illegal  because  collective  bargaining  did  not  take  place. 

B.  illegal  because  the  Conciliation  Commissioner  failed  to  perform 
all  his  duties. 

C.  legal  because  the  award  favoured  the  employers. 

D.  legal  because  either  side  may  reject  an  award  of  a 


Conciliation  Board. 
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36.  Canada  is  to  Metis  as  Latin  America  is  to 

A.  Aztec 

B .  Cree 

C .  Iroquois 

D.  Mestizo 

Another  number  of  items  which  differed  widely  in  categorization  as 
to  the  thought  process  required  but  which  required  knowledge  of  the  same 

i 

content  loaded  upon  the  same  factor.  Items  82  and  85  with  Table  III  il¬ 
lustrate  this  pattern. 

82.  In  an  attempt  to  make  Rhodesia  relax  its  stand  on  Negro 
representation  in  government  all  the  following  pressures 
have  been  imposed  EXCEPT — 

A.  military 

B.  diplomatic 

C.  press 

D.  economic 

85.  I.  The  number  of  votes  in  the  General  Assembly  is  granted 
on  the  basis  of  the  population  of  a  nation. 

II.  The  African  Asian  lands  form  the  largest  block  in 
the  world  organization  when  they  agree  to  vote  together 
for  a  particular  purpose. 

(On  Item  85  students  were  required  to  indicate  the  degree  of  truth 
or  untruth  of  this  combination  of  statements . ) 

It  was  also  noted  that  certain  items  categorized  as  requiring  the 
same  thought  process  and  sampling  the  same  narrow  area  of  content  showed 
widely  differing  factor  patterns.  This  situation  was  not  common  but  did 
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TABLE  II 


SELECTED  FACTOR  LOADINGS  ON 
ITEMS  THIRTY -FIVE  AND  THIRTY-SIX 


Item  Item  Factor  Loadings 


Number 

Category 

Above  . 3 

Level 

(2) 

(3) 

w 

(5) 

35 

1.25 

4157 

5063 

3269 

3827 

3  6 

1.20 

3123 

4493 

4224 

3513 

'  ’  ■  L 1 
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TABLE  III 

SELECTED  FACTOR  LOADINGS  ON 
ITEMS  EIGHTY -TWO  AND  EIGHTY -FIVE 


Item 

Item 

Factor  Loadings 

Number 

Category 

Above  .3  Level 

1) 

(2) 

(3) 

(13) 

82 

1.10 

3232 

4058 

4850 

4894 

85 

4.20 

4643 

^519 

3118 

4208 

■  , 
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occur  with  respect  to  three  of  four  item  combinations.  Items  2  and  3  and 
Table  IV  illustrate  this  lack  of  relationship. 

2.  The  most  extensive  railway  system  in  Iatin  America  is  found  in : 

A.  Argentina 

B.  Brazil 

C .  Chile 

D.  Uruguay 


3-  The  most  industrialized  country  in  South  America  iss 

A.  Argentina 

B .  Brazil 

C .  Colombia 

D.  Venezuela 

Most  of  the  individual  items  showed  factor  loadings  that  could  be 
classified  as  following  one  of  the  first  three  patterns. 


Results  of  Subtest  Factor  Analysis ;  Hypotheses  A2  and  A3*  as  in 
the  case  of  inter-item  analysis,  were  not  supported  by  factor  analysis  of 
groups  of  items  which  were  homogeneous  as  to  the  content  area  they  were 
intended  to  sample. 

One  major  factor  accounted  for  a  large  proportion  of  the  variance 
—forty  per  cent  in  the  case  of  the  unrotated  factor  matrix  for  five 
factors .  Nine  of  the  thirteen  subtest  variables  loaded  consistently  on 
this  major,  seemingly  general  factor. 

Four  of  the  subtests  loaded  heavily  on  other  factors.  In  the  case 
of  the  Quartimax  solution  for  five  factors  shown  in  Table  V,  these  sub¬ 


tests  with  loadings  all  in  excess  of  .7  served  to  define  the  remaining 
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TABLE  IV 

SELECTED  FACTOR  LOADINGS  ON 
ITEMS  TWO  AND  THREE 


Item 

Item 

Factor  Loadings 

Number 

Category 

Above  . 3  Level 

(1)  (22) 

2 

1.12 

6959 

3 

1.12 

5589 

. 
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TABLE  V 

QUARTIMAX  FACTOR  SOLUTION  FOR  SUBTESTS 


Subtests 


Factor  Loadings 


(i) 

(2) 

(3) 

(5) 

1. 

Knowledge  Items ““Geography 
Environment  and  Living 

492 

-024 

-021 

065 

112 

2 . 

Knowledge  Items ““Industrial 
Expansion,  labour  and  Business 

744 

-052 

053 

068 

-189 

3- 

Knowledge  Items -“Development 
of  American  Culture 

765 

1 

0 

0 

VO 

071 

050 

187 

4. 

Knowledge  Items -“Democratic. 
Government  in  Canada 

768 

051 

008 

“114 

078 

5- 

Knowledge  Items — Home  and 
Community  Living 

244 

958 

018 

032 

-016 

6. 

Knowledge  Items — 

Current  Affairs 

723 

029 

-O69 

-099 

024 

7- 

HMP  Items — Geography 
Environment  and  Living 

749 

019 

-060 

-015 

183 

8. 

HMP  Items— Industrial 

Expansion,  labour  and  Business 

724 

129 

-079 

043 

-224 

9- 

HMP  Items --Development  of 
American  Culture 

355 

038 

-003 

221 

051 

10. 

HMP  Items --Democratic 

Government  in  Canada 

653 

-093 

052 

145 

“092 

11. 

HMP  Items- — Home  and 

Community  Living 

342 

018 

919 

-005 

“012 

12. 

HMP  Items-- 
Current  Affairs 

678 

081 

053 

-020 

190 

13- 

HMP  Items — 

Inquiry 

633 

038 

189 

126 

-oi4 

v  : 
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factors.  However,  these  subtests  could  not  be  grouped  as  either  knowledge 
of  higher  mental  process  subtests,  two  being  from  each  process  grouping. 

Other  factor  rotations,  including  an  oblique  solution  based  on 
promax,  did  not  substantially  alter  this  pattern  for  the  subtests. 

II.  CONCURRENT  VALIDITY 


Hypotheses  B1  and  B2 

(Bl)  A  positive  correlation  will  be  obtained  between 
scores  on  objective  test  items  identified  as  re- 
quiring  only  recall  and  the  following  external 
criteria: 

A.  SCAT  Total  Scores 

B.  SCAT  Verbal  Sub-Scores 

C.  SCAT  Quantitative  Sub-Scores 

D.  Principal  Ratings 

E.  Examination  Sub-Scores,  from  the  Essay  Section, 

Part  II  of  The  Examination 

(B2)  A  positive  correlation  will  be  obtained  between 
scores  on  objective  test  identified  as  requiring 
higher  mental  processes  and  the  criteria  presented 
under  Bl  above. 

Hypotheses  Bl  and  B2  were  supported  in  all  cases  at  the  .01  level 
of  significance.  Table  VI  shows  the  correlation  coefficients  obtained 
between  the  knowledge  and  higher  mental  process  subtests  and  the  external 
criteria.  The  support  given  these  hypotheses  provided  a  basis  for 
Hypothesis  B3  which  related  directly  to  the  concurrent  validity  of  the  test. 


' 

CORRELATION  OF  KNOWLEDGE  AND  HIGHER  MENTAL  PROCESS 
SUBTESTS  WITH  EXTERNAL  CRITERIA 
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Hypothesis  B3 

Correlations  obtained  while  testing  Hypothesis  B1  will 

differ  significantly  from  correlations  obtained  while  testing 

Hypothesis  B2. 

The  results  of  testing  Hypothesis  B3  are  reported  as  they  relate 
to  each  of  the  external  criteria.  Table  VII  provides  a  summary  of  these 
results . 

SCAT  Scores  s  Hypothesis  B3  was  not  supported  at  the  .05  level  of 
significance  for  SCAT  verbal,  quantitative  and  total  scores  in  both  the 
Committees 1  and  Judges  classification  of  items,.  Although  the  verbal  and 
total  scores  correlated  more  highly  with  higher  mental  process  subtests, 
the  difference  was  not  statistically  significant. 

Principal’s  Ratings :  Hypothesis  B3  was  supported  at  the  .05  level 
of  significance  in  both  the  committee’s  and  judges'  classification  of 
items  and  their  relationship  to  The  Principal's  Ratings.  Knowledge  item 
subtests  correlated  significantly  higher  with  Principal's  Ratings  than 
higher  mental  process  item  subtests. 

Part  II  Scores ;  Hypothesis  B3  was  supported  at  the  .05  level  of 
significance  for  the  committee^  subtests  and  Part  II  Scores.  For  the 
judges  classification  the  significance  was  very  near  the  .05  level.  In 
all  cases  the  knowledge  subtests  correlated  more  highly  with  the  Part  II 
scores  than  the  higher  mental  process  subtests. 


, 


■ 


. 


56 


TABLE  VII 

SIGNIFICANCE  OF  DIFFERENCES  IN  CORRELATION  BETWEEN 
KNOWLEDGE  AND  HIGHER  MENTAL  PROCESS  SUBTESTS 
IN  RELATION  TO  EXTERNAL  CRITERIA 


SCAT  SCAT  SCAT  PRINCIPAL'S 

VERBAL  QUANTITATIVE  TOTAL  RATINGS 

PART  II 

(6) 

(7) 

(8) 

(9) 

(10; 

2  Knowledge  Item 

Subtest  (lA) 

707 

571 

723 

728 

685 

3  Higher  Mental  Process 
Item  Subtest  (IB) 

729 

560 

731 

640 

605 

Level  of  Significance 

•  3 

•  3 

.3 

.05* 

.05* 

4  Knowledge  Item 

Subtest  (2A) 

714 

565 

725 

725 

680 

5  Higher  Mental  Process 
Item  Subtest  (2B) 

729 

574 

737 

645 

612 

Level  of  Significance 

.3 

.3 

•  3 

.05* 

.1 

* 

Significant  in  relation  to  Hypothesis  B3 
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CHAPTER  V 


SUMMARY,  CONCLUSIONS  AND  SUGGESTIONS  FOR  FURTHER  RESEARCH 

Io  SUMMARY 

The  purpose  of  this  study  wa3  to  attempt  to  find  evidence  that  Fart 
I  of  the  1966  Grade  Nine  Departmental  Examination  in  Social  Studies  did  require 
the  use  of  intellectual  abilities  and  skills  in  a  manner  distinct  from  the 
ability  to  recall  factual  material .  Specifically,  the  relationship  between 
knowledge  items  and  higher  mental  process  items,  individually  and  in  groups, 
was  examined  using  tests  of  construct  and  concurrent  validity. 

The  main  findings  of  the  investigation  may  be  summarized  as  follows: 

1 .  Individual  judges  showed  a  high  proportion  of  agreement  in 
classifying  the  multiple  choice  items  of  Part  I  of  the  exam¬ 
ination  according  to  the  major  levels  of  The  Taxonomy.  With 
the  exception  of  two  items  they  distinguished  between  recall 
and  higher  mental  process  items. 

2.  Judges  showed  considerable  agreement  with  the  initial  classification 
of  the  items.  However,  in  a  number  of  cases  the  judges  showed  a 
tendency  to  classify  items  at  a  lower  level  than  that  originally 
set  by  the  committee  constructing  the  examination. 

3.  Factor  analysis  conducted  on  Part  I,  both  by  individual  items 
and  by  subtests,  revealed  a  large  general  factor  which  account¬ 
ed  for  a  large  proportion  of  the  score  variance. 
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4.  Factor  analysis  revealed  no  clear  evidence  that  the  test  did 
distinguish  between  recall  and  higher  mental  abilities.  It 
seemed  that  a  number  of  uncontrolled  variables,,  particularly 
a  content  relationship,  interfered  with  a  clear  interpreta¬ 
tion  of  results. 

5*  Factor  analysis  did  reveal  that  questions  which  appeared  to 
be  carefully  structured  and  did  not  relate  directly  to  course 
content  did  tend  to  load  uniformly  on  certain  factors. 

6.  Subtests  of  knowledge  or  recall  items  correlated  significant¬ 
ly  higher  with  Principal’s  Ratings  and  Part  II  scores  than 
did  subtests  composed  of  higher  mental  process  items. 

7*  Subtests  of  knowledge  or  recall  items  and  subtests  of  higher 
mental  process  items  did  not  show  a  significant  difference  in 
correlations  with  SCAT  scores,  although  a  tendency  for  the 
verbal  subscore  of  the  SCAT  test  to  correlate  more  highly 
with  the  higher  mental  process  subtests  was  evident. 

II.  CONCLUSIONS  AND  IMPLICATIONS 

It  is  convenient  to  group  the  contents  of  this  section  under  two 
headings,  corresponding  to  the  two  groups  of  hypotheses. 

Construct  Validity 

It  would  appear  that  when  The  Taxonomy  serves  as  a  basis  for 
constructing  items  to  measure  the  behaviour  described  therein,  judges 
can  assign  the  items  to  appropriate  categories  with  some  accuracy.  lack 
of  agreement,  when  it  occurs  among  themselves,  or  with  the  a  priori 
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classification  can  seemingly  be  ascribed  to  at  least  the  following  two 
reasons,  both  of  which  have  a  common  basis  in  that  the  item  writer  and 
item  judge  have  different  conceptions  of  the  competencies  and  problem 
solving  methods  used  by  students  for  whom  the  items  were  intended.  First 
the  item  writers  might  have  prepared  an  item  for  an  upper  level  of  The 
Taxonomy  and  the  rater  believed  the  student  will  know  the  answer  to  the 
item  because  of  prior  knowledge,  as  reflected  in  the  downward  trend  in 
the  judges  identifying  of  test  iems.  Second,  the  item  writers  might 
have  prepared  an  item  to  evoke  behaviours  regarded  as  "evaluation",  but 
the  student  might  have  answered  the  question  by  systematically  elimina¬ 
ting  distractors  or  some  other  method.  Thus,  the  intended  process  dif¬ 
fered  from  the  obtained  process. 

Identification  by  judges  was  of  no  empirical  significance,  being 
very  subjective  in  nature,  but  it  did  provide  an  assessment  of  the  ini¬ 
tial  classification  on  which  later  statistical  analysis  was  based. 

As  other  studies  have  shown,  factor  analysis  revealed  a  large  or 
major  factor  which  could  be  interpreted  as  being  one  representing  generax 
competence  and  mastery  of  course  content.  Further  interpretation  of 
these  results  in  a  meaningful  and  comprehensive  manner  was  impossible 
due,  it  would  seem,  to  several  other  variables  affecting  the  factor 
patterns . 

A  content  variable  was  thought  to  be  a  major  source  of  the  inter¬ 
ference.  The  test  sampled  a  wide  area  of  content  and  consequently  stu¬ 
dent  familiarity  with  and  competence  on  each  question  would  differ  ac¬ 
cording  to  his  interest  in  and  mastery  of  each  content  area. 

A  "teaching"  or  instructional  variable  was  thought  to  be  another 
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source  of  interference.  Teachers  would  probably  teach  students  to  ap¬ 
proach  questions  in  ways  unanticipated  by  the  item  writers.  In  this  type 
of  testing  situation  there  is  no  appraisal  of  the  way  in  which  an  answer 
is  derived. 

Also  contributing  to  the  inconclusiveness  of  the  test  results  was 
probably  the  overall  hetereogeneity  of  the  testing  situation. 

Results  did  suggest,  however,  that  in  cases  where  the  item  was 
intended  to  evoke  higher  mental  processes  and  the  content  base  was 
structured  independent  of  course  content,  then  evidence  of  some  "higher" 
intellectual  process  could  be  recognized. 

Concurrent  Validity 

Empirical  evidence  in  support  of  concurrent  validity  indicated  that 
knowledge  items  did  require  a  type  or  level  of  ability  different  from  that 
of  higher  mental  process  items. 

However,  since  the  construct  validity  of  the  test  was  not  proven  in 
this  regard  it  can  not  be  definitely  stated  what  factor  accounted  for  this 
difference  in  relating  to  external  criteria.  It  may  be  that  the  knowledge 
or  recall  items  were  much  more  closely  allied  to  course  content  in  design 
than  were  the  higher  mental  process  items  or  it  may  be  that  different 
abilities  were  actually  required  or  a  combination  of  factors  could  have 
been  operating. 

Evidence  would  also  seem  to  indicate  that  the  teachers '  evalua¬ 
tion  of  their  students  both  in  the  form  of  a  general  rating  and  in  the 
form  of  essay  and  paragraph  marks  is  allied  more  closely  with  the  recall 
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III.  SUGGESTIONS  FOR  FURTHER  RESEARCH 

While  not  supporting  the  hypotheses  and  answering  conclusively 
the  basic  problem  of  the  study,  the  results  did  point  out  several  pos¬ 
sibilities  for  further  constructive  research. 

This  study  has  shown  something  of  the  complexity  of  evaluating  a 
process  dimension  of  learning  in  a  mass  testing  situation.  Further  re¬ 
search  is  needed  both  in  the  selection  of  examination  questions  and  in 
the  analysis  of  examination  results.  There  is  a  possibility  that  the 
utilization  of  sophisticated  item  selection  technique  such  as  that 

developed  by  Wahlstrom’1'  and  the  application  of  new  techniques  of  factor 

2 

analysis  such  as  that  developed  by  Hendrickson  and  White  would  allow 
for  meanginful  interpretation  of  large  scale  examination  results  in 
terms  of  the  levels  of  The  Taxonomy. 

Results  of  factor  analysis  suggested  that  several  variables  need 
to  be  studied  and  controlled  if  intellectual  abilities  and  skills  are  to 
be  assessed.  To  validate  The  Taxonomy  there  is  a  need  for  more  studies 
in  which  the  teaching  technique  is  known  and  oontrolled,  the  content  area 
strictly  limited,  and  the  testing  instrument  carefully  constructed. 
Studies  of  this  nature  will  be  needed  in  all  subject  areas  as  a  basis  for 
applying  The  Taxonomy  with  clarity  and  meaningfulness  to  large  scale 
testing  situations. 


Merlin  E.  Wahlstrom  "A  Factor  Analytic  Technique  for  Item 
Selection.”  (unpublished  Ph.  D.  Thesis,  The  University  of  Alberta,  1967). 

^Alan  E.  Hendrickson  and  Paul  D.  White.  "A  Method  for  the 
Rotation  of  Higher  Order  Factors."  The  British  Journal  of  Mathematical 
and  Statistical  Psychology  (Vol.  XIX,  Part  I;  1966) . 
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The  responses  of  the  judges  and  the  results  of  factor  analysis  and 

intercorrelation  imply  the  need  for  study  of  the  actual  learning  situation 

3 

which  test  results  are  intended  to  reflect.  Taba  has,  among  others, 
pointed  out  that  tests  and  test  results  should  be  part  of  a  complete  and 
well  planned  process.  If  intellectual  abilities  and  skills  are  to  be 
assessed  then  the  teaching  must  show  such  evidence  that  it  provides  for 
the  acquisition  of  these  abilities  and  skills. 

Further  study  is  needed  of  the  predictive  qualities  of  examina¬ 
tions  intended  to  assess  higher  mental  processes  and  their  interrelation¬ 
ship  with  other  forms  of  evaluation.  There  is  an  evident  difference  in 
the  relationship  of  knowledge  and  higher  mental  process  questions  to  other 
evaluative  criteria  but  the  reasons  for  this  difference  are  not  known. 

Do  teachers  conceive  of  student  ability  in  terms  of  their  ability  to 
master  content  and  recall  material?  Do  written  examinations  require  the 
utilization  of  intellectual  abilities  and  skills  and  are  these  actually 
assessed  in  the  evaluation  procedures  used?  What  is  the  actual  relation¬ 
ship  of  standardized  tests  to  external  subject  area  examinations?  All 
these  questions  need  reappraisal  in  terms  of  the  entry  of  a  process  dimen¬ 
sion  into  testing  based  on  The  Taxonomy. 

The  use  of  The  Taxonomy  as  developed  by  Bloom  and  his  associate  s 
has  brought  a  general  consciousness  of  the  need  to  study  the  whole 
problem  of  assessing  objective  attainment  in  the  social  studies  and  in 
particular  to  study  the  problem  of  developing  and  evaluating  intellectual 
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abilities  and  skills  other  than  recall.  Present  testing  procedures  do 
not  allow  for  a  statistical  verification  of  this  dimension  but  the 
results  of  this  study  suggest  the  problems  are  not  with  the  theory  itself 
but  with  the  implementation  of  the  theory  in  a  mass  testing  situation. 
Further  research  of  the  type  mentioned  would  serve  to  eliminate  these 


problems . 
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APPENDIX  A 


CONDENSED  VERSION  OF  THE  TAXONOMY 
OF  EDUCATIONAL  OBJECTIVES 
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TO 


COGNITIVE  DOMAIN 

KNOWLEDGE 


IcOO  KNOWLEDGE 

Knowledge,  as  defined  here,  involves  the  recall  of  specifics 
and  universals,  the  recall  of  methods  and  processes ,  or  the  recall 
of  a  pattern,  structure,  or  setting.  For  measurement  purposes,  the 
recall  situation  involves  little  more  than  bringing  to  mind  the 
appropriate  material.  Although  some  alteration  of  the  material  may 
be  required,  this  is  a  relatively  minor  part  of  the  task.  The 
knowledge  objectives  emphasize  most  the  psychological  processes  of 
remembering o  The  process  of  relating  is  also  involved  in  that  a 
knowledge  test  situation  requires  the  organization  and  reorganiza- 
tion  of  a  problem  such  that  it  will  furnish  the  appropriate  signals 
and  cues  for  the  information  and  knowledge  the  individual  possesses. 
To  use  an  analogy,  if  one  thinks  of  the  mind  as  a  file,  the  problem 
in  a  knowledge  test  situation  is  that  of  finding  in  the  problem  or 
task  the  appropriate  signals,  cues,  and  clues  which  will  most  ef¬ 
fectively  bring  out  whatever  knowledge  is  filed  or  stored. 

1.10  KNOWLEDGE  OF  SPECIFICS 

The  recall  of  specific  and  isolable  bits  of  information.  The 
emphasis  is  on  symbols  with  concrete  referents.  This  material, 
which  is  at  a  very  low  level  of  abstraction,  may  be  thought  of  as 
the  elements  from  which  more  complex  and  abstract  forms  of  know- 
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ledge  are  built. 


1.11  KNOWLEDGE  OF  TERMINOLOGY 

Knowledge  of  the  referents  for  specific  symbols  (verbal  and 
non-verbal).  This  may  include  knowledge  of  the  most  generally 
accepted  symbol  referent,  knowledge  of  the  variety  of  symbols  which 
may  be  used  for  a  single  referent,  or  knowledge  of  the  referent 
most  appropriate  to  a  given  use  of  a  symbol. 

*To  define  technical  terms  by  giving  their  attributes, 
properties ,  or  relations . 

*Familiarity  with  a  large  number  of  words  in  their  common 
range  of  meanings. 


1.12  KNOWLEDGE  OF  SPECIFIC  FACTS 

Knowledge  of  dates,  events,  persons,  places,  etc.  This  may 
include  very  precise  and  specific  information  such  as  the  specific 
date  or  exact  magnitude  of  a  phenomenon.  It  may  also  include 
approximate  or  relative  information  such  as  an  approximate  time 
period  or  the  general  order  of  magnitude  of  a  phenomenon 
*The  recall  of  major  facts  about  particular  cultures . 

*The  possession  of  a  minimum  knowledge  about  the  organisms 
studied  in  the  laboratory. 


1.20  KNOWLEDGE  OF  WAYS  AND  MEANS  OF  DEALING  WITH  SPECIFICS 


Knowledge  of  the  ways  of  organizing,  studying,  judging  and 
criticizing.  This  includes  the  methods  of  inquiry,  the  chronologi¬ 
cal  sequences,  and  the  standards  of  judgment  within  a  field  as  well 
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as  the  patterns  of  organization  through  which  the  areas  of  the 
fields  themselves  are  determined  and  internally  organized.  This 
knowledge  is  at  an  intermediate  level  of  abstraction  between 
specific  knowledge  on  the  one  hand  and  knowledge  of  universals  on 
the  other.  It  does  not  so  much  demand  the  activity  of  the  student 
in  using  the  materials  as  it  does  a  more  passive  awareness  of  their 
nature. 

1.21  KNOWLEDGE  OF  CONVENTIONS 

Knowledge  of  characteristic  ways  of  treating  and  presenting 
ideas  and  phenomena.  For  purposes  of  communication  and  consistency, 
workers  in  a  field  employ  usages,  styles,  practices,  and  forms 
which  best  suit  their  purposes  and/or  which  appear  to  suit  best  the 
phenomena  with  which  they  deal.  It  should  be  recognized  that  al¬ 
though  these  forms  and  conventions  are  likely  to  be  set  up  on 
arbitrary,  accidental,  or  authoritative  bases,  they  are  retained 
because  of  the  general  agreement  or  concurrence  of  individuals  con¬ 
cerned  with  the  subject,  phenomena,  or  problem. 

^Familiarity  with  the  forms  and  conventions  of  the  major 
types  of  works,  e.g.,  verse,  plays,  scientific  papers,  etc. 

*To  make  pupils  conscious  of  correct  form  and  usage  in 
speech  and  writing. 

1.22  KNOWLEDGE  OF  TRENDS  AND  SEQUENCES 

Knowledge  of  the  processes,  directions,  and  movements  of 
phenomena  with  respect  to  time. 
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Understanding  the  continuity  and  development  of  American 
culture  as  exemplified  in  American  Life. 

*Khowledge  of  the  basic  trends  underlying  the  development 
of  public  assistance  programs. 

1.23  KNOWLEDGE  OF  CLASSIFICATIONS  AND  CATEGORIES 

Knowledge  of  the  classes,  sets,  division,  and  arrangements 
which  are  regarded  as  fundamental  for  a  given  subject  field,  pur¬ 
pose,  argument,  or  problem. 

*To  recognize  the  area  encompassed  by  various  kinds  of 
problems  or  materials. 

^Becoming  familiar  with  a  range  of  types  of  literature. 

1.24  KNOWLEDGE  OF  CRITERIA 

Knowledge  of  the  criteria  by  which  facts,  principles, 
opinions,  and  conduct  are  tested  or  judged. 

*Familiarity  with  criteria  for  judgment  appropriate  to  the 
type  of  work  and  the  purpose  for  which  it  is  read. 

^Knowledge  of  criteria  for  the  evaluation  of  recreational 
activities . 

1.25  KNOWLEDGE  OF  METHODOLOGY 

Knowledge  of  the  methods  of  inquiry,  techniques,  and  pro¬ 
cedures  employed  in  a  particular  subject  field  as  well  as  those  em¬ 
ployed  in  investigating  particular  problems  and  phenomena.  The 
emphasis  here  is  on  the  individual's  knowledge  of  the  method. 


. 
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*Know.ledge  of  scientific  methods  for  evaluating  health  con¬ 
cepts  . 

*The  student  shall  know  the  methods  of  attack  relevant  to 
the  kinds  of  problems  of  concern  to  the  social  sciences. 

1-30  KNOWLEDGE  OF  THE  UNIVERSALS  AND  ABSTRACTIONS  IN  A  FIELD 

Knowledge  of  the  major  schemes  and  patterns  by  which  phen¬ 
omena  and  ideas  are  organized.  These  are  the  large  structures, 
theories,  and  generalizations  which  dominate  a  subject  field  or 
which  are  quite  generally  used  in  studying  phenomena  or  solving 
problems.  These  are  at  the  highest  levels  of  abstraction  and 
complexity. 

1.31  KNOWLEDGE  OF  PRINCIPLES  AND  GENERALIZATIONS 

Knowledge  of  particular  abstractions  which  summarize  observa¬ 
tions  of  phenomena.  These  are  the  abstractions  which  are  of  value 
in  explaining,  describing,  predicting,  or  in  determining  the  most 
appropriate  and  relevant  action  or  direction  to  be  taken. 

•^Knowledge  of  the  important  principles  by  which  our  exper¬ 
ience  with  biological  phenomena  is  summarized. 

*The  recall  of  major  generalizations  about  particular 
cultures . 

1.32  KNOWLEDGE  OF  THEORIES  AND  STRUCTURES 

Knowledge  of  the  Body  of  principles  and  generalizations  to¬ 
gether  with  their  interrelations  which  present  a  clear,  rounded,  and 
systematic  view  of  a  complex  phenomenon,  problem,  or  field.  These 
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are  the  most  abstract  formulations,  and  they  can  be  used  to  show 
the  interrelation  and  organization  of  a  great  range  of  specifics. 

*The  recall  of  major  theories  about  particular  cultures . 

^Knowledge  of  a  relatively  complete  formulation  of  the 
theory  of  evolution. 

HIGHER  MENTAL  PROCESSES 

Abilities  and  skills  refer  to  organized  modes  of  operation  and 
generalized  techniques  for  dealing  with  materials  and  problems.  The 
materials  and  problems  maybe  of  such  a  nature  that  little  or  no  special¬ 
ized  and  technical  information  is  required.  Such  information  as  is  re¬ 
quired  can  be  assumed  to  be  part  of  the  individual's  general  fund  of  know¬ 
ledge.  Other  problems  may  require  specialized  and  technical  information 
at  a  rather  high  level  such  that  specific  knowledge  and  skill  in  dealing 
with  the  problem  and  the  materials  are  required.  The  abilities  and  skill 
objectives  emphasize  the  mental  processes  of  organizing  and  reorganizing 
material  to  achieve  a  particular  purpose.  The  materials  may  be  given  or 
remembered. 

2.00  COMPREHENSION 

This  represents  the  lowest  level  of  understanding.  It  refers 
to  a  type  of  understanding  or  apprehension  such  that  the  individual 
knows  what  is  being  communicated  and  can  make  use  of  the  material 
or  idea  being  communicated  without  necessarily  relating  it  to  other 
material  or  seeing  its  fullest  implications . 
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2.10  TRANSLATION 

Comprehension  as  evidenced  by  the  care  and  accuracy  with 
which  the  communication  is  paraphrased  or  rendered  from  one  langu¬ 
age  or  form  of  communication  to  another.  Translation  is  judged  on 
the  basis  of  faithfulness  and  accuracy,  that  is,  on  the  extent  to 
which  the  material  in  the  original  communication  is  preserved  al¬ 
though  the  form  of  the  communication  has  been  altered. 

*The  ability  to  understand  non-literal  statements  (metaphor, 
symbolism,  irony,  exaggeration) . 

*Ski.ll  in  translating  mathematical  verbal  material  into 
symbolic  statements  and  vice  versa. 

2.20  INTERPRETATION 

The  explanation  or  summarization  of  a  communication.  Whereas 
translation  involves  an  objective  part -for -part  rendering  of  a  com¬ 
munication,  interpretation  involves  a  reordering,  rearrangement,  or 
a  new  view  of  the  material. 

*The  ability  to  grasp  the  thought  of  the  work  as  a  whole  at 
any  desired  level  of  generality. 

*The  ability  to  interpret  various  types  of  social  data. 

2.30  EXTRAPOLATION 

The  extension  of  trends  or  tendencies  beyond  the  given  data 
to  determine  implications,  consequences,  corollaries,  effects,  etc., 
which  are  in  accordance  with  the  conditions  described  in  the 


original  communication. 
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*The  ability  to  deal  with  the  conclusions  of  a  work  in  terms 
of  the  immediate  inference  made  from  the  explicit  state¬ 
ments  . 

*Skills  in  predicting  continuation  of  trends. 

3-00  APPLICATION 

The  use  of  abstractions  in  particular  and  concrete  situa¬ 
tions.  The  abstractions  may  be  in  the  form  of  general  ideas,  rules 
of  procedures,  or  generalized  methods.  The  abstractions  may  also 
be  technical  principles,  ideas,  and  theories  which  must  be  re¬ 
membered  and  applied. 

^Application  to  the  phenomena  discussed  in  one  paper  of  the 
scientific  terms  or  concepts  used  in  other  papers. 

*The  ability  to  predict  the  probable  effect  of  a  change  in 
a  factor  on  a  biological  situation  previously  at  equili¬ 
brium. 

4.00  ANALYSIS 

The  breakdown  of  a  communication  into  its  constituent  ele¬ 
ments  or  parts  such  that  the  relative  hierarchy  of  ideas  is  made 
clear  and/or  the  relations  between  the  ideas  expressed  are  made 
explicit.  Such  analyses  are  intended  to  clarify  the  communication, 
to  indicate  how  the  communication  is  organized,  and  the  way  in 
which  it  manages  to  convey  its  effects,  as  well  as  its  basis  and 
arrangement . 
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4.10  ANALYSIS  OF  ELEMENTS 

Identification  of  the  elements  included  in  a  communication. 
*The  ability  to  recognize  unstated  assumptions. 

*Skills  in  distinguishing  facts  from  hypotheses. 

4.20  ANALYSES  OF  REIATIONSHIPS 

The  connections  and  interactions  between  elements  and  parts 
of  a  communication. 

^Ability  to  check  the  consistency  of  hypotheses  with  given 
information  and  assumptions. 

*Skills  in  comprehending  the  interrelationships  among  the 
ideas  in  a  passage. 


4.30  ANALYSIS  OF  ORGANIZATIONAL  PRINCIPLES 

The  organization,  systematic  arrangement ,  and  structure 
which  hold  the  communication  together.  This  includes  the  '‘explicit" 
as  well  as  "implicit"  structure.  It  includes  the  bases,  necessary 
arrangement,  and  the  mechanics  which  make  the  communication  a  unit. 
*The  ability  to  recognize  form  and  pattern  in  literary  or 
artistic  works  as  a  means  of  understanding  their  meaning. 
*Ability  to  recognize  the  general  techniques  used  in  per¬ 
suasive  materials,  such  as  advertising,  propaganda,  etc. 


5.00  SYNTHESIS 

The  putting  together  of  elements  and  parts  so  as  to  form  a 


whole.  This  involves  the  process  of  working  with  pieces,  parts. 
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elements,  etc»,  and  arranging  and  combining  them  in  such  a  way  as 
to  constitute  a  pattern  or  structure  not  clearly  there  before. 

5-10  PRODUCTION  OF  A  UNIQUE  COMMUNICATION 

The  development  of  a  communication  in  which  the  writer  or 
speaker  attempts  to  convey  ideas,  feelings,  and/or  experiences  to 
others . 

*Skill  in  writing,  using  an  excellent  organization  of  ideas 
and  statements. 

*Ability  to  tell  a  personal  experience  effectively. 

5*20  PRODUCTION  OF  A  PLAN,  OR  PROPOSED  SET  OF  OPERATIONS 

The  development  of  a  plan  of  work  or  the  proposal  of  a  plan 
of  operations o  The  plan  should  satisfy  requirements  of  the  task 
which  may  be  given  to  the  student  or  which  he  may  develop  himself. 

*Ability  to  propose  ways  of  testing  hypotheses. 

*Ability  to  plan  a  unit  of  instruction  for  a  particular 
teaching  situation. 

5-30  DERIVATION  OF  A  SET  OF  ABSTRACT  RELATIONS 

The  development  of  a  set  of  abstract  relations  either  to 
classify  or  explain  particular  data  or  phenomena,  or  the  deduction 
of  propositions  and  relations  from  a  set  of  basic  propositions  or 
symbolic  representations. 

^Ability  to  formulate  appropriate  hypotheses  based  upon  an 
analysis  of  factors  involved,  and  to  modify  such  hypotheses 
in  the  light  of  new  factors  and  considerations. 
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*Ability  to  make  mathematical  discoveries  and  generalizations . 

6,00  EVALUATION 

Judgments  about  the  value  of  material  and  methods  for  given 
purposes.  Quantitative  and  qualitative  judgments  about  the  extent 
to  which  material  and  methods  satisfy  criteria.  Use  of  a  standard 
of  appraisal.  The  criteria  may  be  those  determined  by  the  student 
or  those  which  are  given  to  him. 

6,10  JUDGMENTS  IN  TERMS  QF  INTERNAL  EVIDENCE 

Evaluation  of  the  accuracy  of  a  communication  from  such 
evidence  as  logical  accuracy,  consistency,  and  other  internal 
criteria . 

^Judging  by  internal  standards,  the  ability  to  assess  general 
probability  of  accuracy  in  reporting  facts  from  the  care 
given  to  exactness  of  statement,  documentation,  proof,  etc. 

*The  ability  to  indicate  logical  fallacies  in  arguments. 

6.20  JUDGMENTS  IN  TERMS  OF  EXTERNAL  CRITERIA 

Evaluation  of  material  with  reference  to  selected  or  remem¬ 
bered  criteria. 

•fr^The  comparison  of  major  theories,  generalizations,  and  facts 
about  particular  cultures . 

^Judging  by  external  standards,  the  ability  to  compare  a  work 
with  the  highest  known  standards  in  its  field— especially 
with  other  works  of  recognized  excellence. 
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Item  No. 

TL 

BC 

Diff 

Item  No. 

TL 

BC 

Diff 

1 

1.20 

472 

53 

4l 

1.12 

439 

68 

2 

1.12 

222 

47 

42 

1.10 

501 

73 

3 

1.12 

382 

59 

43 

1.22 

634 

74 

4 

1.10 

500 

66 

44 

1.12 

419 

53 

5 

1.10 

307 

65 

45 

1.22 

539 

57 

6 

1.10 

403 

80 

46 

1.22 

567 

70 

7 

2.20 

098 

35 

47 

1.22 

4i6 

48 

8 

3-00 

228 

54 

48 

1.10 

475 

56 

9 

3-00 

409 

68 

49 

2.20 

407 

32 

10 

2.20 

417 

72 

50 

4.20 

321 

58 

11 

2.20 

472 

56 

51 

4.20 

298 

62 

12 

3-00 

430 

69 

52 

4.20 

350 

70 

13 

3.00 

585 

60 

53 

2.20 

139 

4o 

14 

3-00 

446 

4l 

54 

2.20 

174 

44 

15 

2.20 

362 

60 

55 

2.20 

220 

28 

16 

2.20 

380 

78 

56 

1.22 

427 

61 

17 

2.20 

528 

64 

57 

1.22 

510 

60 

18 

2.20 

573 

51 

58 

1.10 

572 

55 

19 

2.20 

028 

12 

59 

1.10 

434 

53 

20 

2.20 

674 

52 

60 

1.10 

504 

77 

21 

1.22-4 .20 

281 

52 

61 

1.10 

471 

77 

22 

1.25-4.00 

613 

51 

62 

1.20 

663 

78 

23 

1.20 

521 

73 

63 

1.31-2.1 

518 

60 

24 

1.20 

485 

42 

64 

3.00-4.00 

378 

55 

25 

1.20 

393 

57 

65 

3.00-4.00 

393 

53 

26 

1.22 

438 

47 

66 

3.00-4.00 

474 

45 

27 

2.20-3.00 

483 

86 

67 

3.00-4.00 

433 

55 

28 

2.00 

254 

57 

68 

4.00 

380 

65 

29 

2.00-4.00 

536 

46 

69 

4.00 

454 

55 

30 

2.00-4.00 

457 

64 

70 

4.00 

534 

60 

31 

2.00-4.00 

218 

67 

71 

2.20 

437 

88 

32 

3.00 

339 

75 

72 

2.20 

205 

66 

33 

2.30 

528 

40 

73 

2.20 

417 

4l 

34 

2.20 

593 

49 

74 

2.10 

304 

26 

35 

1.25 

484 

78 

75 

2.30 

281 

47 

36 

1.20 

569 

79 

76 

1.10 

432 

89 

37 

1.00 

152 

70 

77 

1.10 

347 

58 

38 

1.10 

545 

70 

78 

1.12 

574 

70 

39 

1.12 

500 

72 

79 

1.12 

568 

73 

40 

1.12 

535 

73 

80 

1.10 

482 

89 

55 
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Item  No. 

TL 

BC 

Diff 

Item  No. 

TL 

BC 

Diff 

81 

1.10 

349 

23 

91 

2.00 

490 

68 

82 

1.10 

587 

46 

92 

4.00 

060 

67 

83 

4.20 

500 

46 

93 

4.00 

458 

60 

84 

4.20 

487 

64 

94 

4.00 

587 

93 

85 

4.20 

549 

53 

95 

4.00 

497 

65 

86 

4.20 

44l 

27 

96 

6.00 

338 

59 

87 

2.00 

470 

24 

97 

6.00 

333 

32 

88 

2.00 

023 

13 

98 

6.00 

439 

22 

89 

2.00 

510 

66 

99 

6.00 

534 

60 

90 

2.00 

<DO“ 

100 

6.00 

312 

32 

*Item  90  was  eliminated  from  the  test  and  did  not  figure  in  the 
results . 
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:em 

fc> . 

Committee 

TL 

1 

Judges 

2 

-  Bloom's 

3  4 

1 

1.20 

12 

2 

1.12 

12 

3 

1.12 

12 

4 

1.10 

12 

5 

1.10 

12 

6 

1.10 

12 

7 

2.20 

4 

4 

4 

8 

3-00 

2 

10 

9 

3-00 

1 

11 

10 

2.20 

2 

10 

n 

2.20 

1 

10  1 

12 

3-00 

10  2 

13 

3-00 

1 

4 

7 

14 

3-00 

2 

9  1 

15 

2.20 

12 

l6 

2.20 

7 

5 

17 

2.20 

12 

18 

2.20 

12 

19 

2.20 

12 

20 

2.20 

12 

21 

1.22=4.20 

12 

22 

1.25=4.00 

9 

3 

23 

1.20 

11 

1 

24 

1.20 

10 

2 

25 

1.20 

11 

1 

26 

1.22 

6 

5 

1 

27 

2.20=3.00 

3 

9 

28 

3-00 

l 

2 

9 

29 

2.00=4.00 

2 

10 

30 

2.00=4.00 

2 

10 

31 

2.00=4.00 

1 

10 

1 

32 

3-00 

1 

9 

l 

33 

2.30 

9 

2  1 

34 

2.20 

10 

2 

35 

1.25 

1 

9 

2 

Levels 
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Item  Committee  Judges  -  Bloom's  Levels 

No.  TL  12  3^56 


36 

1.20 

9 

1 

37 

1.00 

12 

38 

1.10 

12 

39 

1.12 

10 

1 

40 

1.12 

12 

4l 

1.12 

12 

42 

1.10 

12 

43 

1.22 

10 

2 

44 

1.12 

12 

45 

1.22 

9 

3 

46 

1.22 

11 

1 

47 

1.22 

12 

48 

1.10 

12 

49 

2.20 

11 

1 

50 

4.20 

7 

51 

4.20 

7 

52 

4.20 

7 

53 

2.20 

1 

11 

54 

2.20 

11 

55 

2.20 

11 

56 

1.22 

12 

57 

1.22 

12 

58 

1.10 

12 

59 

1.10 

12 

60 

1.10 

12 

61 

1.10 

12 

62 

1.20 

12 

63 

1.31-2.10 

9 

3 

64 

3.00=4.00 

65 

3.00=4.00 

66 

3.00=4.00 

67 

3.00=4.00 

68 

4.00 

10 

69 

4.00 

1 

8 

70 

4.00 

8 

1 


1 


9 

9 

9 

9 


l 


5 

5 

5 

1 

1 


3 

3 

3 

3 

2 

2 

4 


1 
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Item 

Committee 

Judges  -  Bloom’s  Levels 

No. 

TL 

1 

2345 

6 

71 

2.20 

.11 

1 

72 

2.20 

11 

l 

73 

2.20 

11 

1 

74 

2.10 

9 

2 

l 

75 

2.30 

10 

1 

1 

76 

1.10 

12 

77 

1.10 

12 

78 

1.12 

12 

79 

1.12 

12 

80 

1.10 

12 

81 

1.10 

12 

82 

1.10 

1 2 

83 

4.20 

11 

l 

84 

4.20 

11 

1 

85 

4.20 

11 

l 

86 

4.20 

11 

l 

87 

2.00 

9 

l 

2 

88 

2.00 

9 

l 

2 

89 

2.00 

9 

l 

1 

1 

90 

2.00 

9 

l 

1 

1 

91 

2.00 

9 

2 

l 

92 

4.00 

3 

3 

2 

2 

2 

93 

4.00 

3 

3 

2 

2 

2 

94 

4.00 

3 

3 

2 

2 

2 

95 

6.00 

1 

4 

2 

5 

96 

6.00 

1 

4 

2 

5 

97 

6.00 

2 

3 

3 

4 

98 

6.00 

1 

2 

4 

5 

99 

6.00 

1 

2 

2 

4 

3 

100 

6.00 

4 

4 

2 

2 
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